The Long-Tail of Idea Generation

TeamTrack is used by companies around the world in a myriad of different ways. In fact Serena knows of over 600 different applications that have been created using TeamTrack and they estimate that there are hundreds if not thousands more that they do not know about.
In this series of articles we will look in detail at some of the applications that Client Solutions has delivered to customers in Ireland using Serena TeamTrack.


The Idea Forum Application
An article entitled ‘Big Results from Small Ideas’ by Alan G. Robinson and Dean M. Schroeder looks at the impact of idea generation schemes in companies. They examined over 150 companies in 17 countries and surprisingly found, that the companies that did best were those that implemented lots of small ideas rather than a few big ones. They said,


“We compared the best idea systems in the world--those implementing 20, 50 and even 100 ideas per employee per year--with medium- and low-performing systems. The purpose was to document what works to promote idea generation, what doesn't, and why. One of the most surprising findings of the ‘Ideas Are Free’ study was how high-performing companies focused on small ideas while low-performing companies tended to go after big ones.”


What we are talking about here is the long-tail of idea generation. It is better to focus on lots of small ideas than a couple of big ones. The value of all those small ideas combined together far exceeds the value of a few big ones. In the graph below, we can see that the combined value of the ‘big’ (green) ideas is less than the combined value of the ‘small’ (blue) ideas.




The difficulty with any long tail however, is that the amount of effort expended in gathering and evaluating each idea in the long-tail can outweigh the benefit to be derived from implementing them. That is why you need to automate the process.


Client Solution’s Idea Forum application allows employees throughout an organisation to submit innovative ideas that are then peer reviewed. The top ideas each month or quarter are escalated to management who will consider them for implementation.


The main advantage of this application is that it captures valuable innovative and creative suggestions throughout the organisation. By exposing them to other employees for consideration, the organisation ensures that the best ideas bubble to the top and are improved upon by a forum type discussion. The result is that creative energy is generated by all the ideas and comments on ideas, bouncing off each other.


One good idea can be more than enough to pay for the application and by giving those who are closest to your products or services a voice; who knows what other innovative opportunities can be harvested.


How does it work?

1. An idea is submitted into TeamTrack through a website on the company intranet
2. The author of the idea may take a few days to fine tune the details of the idea and attach images and various bits of documentation to it
3. When the author is ready to have it peer reviewed, she clicks the ‘Send for Peer Review’ button.
4. Users who have ‘peer review’ privileges, can browse the list of ideas that are available for review. They can click into an idea, read the associated documentation, add their own comments and then give the idea a rating.
5. The ideas with the highest average rating or number of reviews bubble to the top.
6. A manager moves all ideas out of the peer review state every quarter and the top ideas are put before management for consideration. The authors of the top rated ideas are rewarded in some way, usually by giving them a percentage of the savings or profit resulting from the idea being implemented.


If you would like to discuss implementing the Idea Forum application in your organisation then please get in touch.


In future articles I will be looking at some other Client Solutions applications developed in TeamTrack that Irish companies are capitalising on.

Using Monte Carlo Simulation to derive more accurate project duration estimates

Monte Carlo Simulation is a relatively new technique in the field of mathematics that uses simulation rather than precise analytical calculations to arrive at an estimated value. In essence it introduces noise into a mathematical analysis. In the real world, actual outcomes seldom match predicted outcomes that are based on probabilities. ‘Noise’ or unanticipated influences often encroach on a situation, causing spurious results.

In this article I am going to look at how Monte Carlo Simulation could be used to give more accurate project duration estimates.

No project runs entirely uninterrupted from start to finish. Team members become ill or other higher priority tasks arise and require immediate attention. This is the ‘noise’ that I am referring to. ‘Noise’ is basically all the things that interfere with the normal running of a project. By looking back at the accuracy of a team’s estimates in the past and injecting random noise, the hope is that a more accurate estimate can be attained.

Forecasting how long a project is going to take has been a near impossible task for companies for years. Few large infrastructure projects where there is a team and strict deadlines in place, are completed on time.

Hugh W. Ryan, in an article for Outlook Journal, summarized research that showed:

- Only 8 percent of applications projects costing between $6 million and $10 million succeed.
- Among all IT development projects, only 16% are delivered to acceptable cost, time and quality.
- Cost overruns for IT projects have been estimated at $59 billion in the United States alone.

According to the Economist, a £456m ($844m) project for Britain's Child Support Agency came in over a year late, and has failed to deliver payments to more than half of eligible applicants and according to Cuppan approximately 80% of the medium to large software projects executed within the Department of Defense (DoD) were 100% over budget and 90% were at least one year behind schedule.

As the main cost in software development is developers, time over-runs quickly translate into cost over-runs. The accurate calculation of how long a project is going to take is vital to the success of a project. With a team of 20 developers, for example, that is costing €30,000 to €40,000 per week, delays soon add up.

In looking for an example of a large project that has run way over original estimation, consider the repeated delays of Microsoft’s Vista operating system. Vista was intended to replace Windows XP as Microsoft’s main operating system. While it is ambitious in terms of what it hoped to achieve (new file system structures and interface standards), it has become something of a joke in the industry. Every few months, Microsoft came out with a new vague release date intention which was ultimately not met.

When asked about Vista during an interview in Sept 2003, Microsoft Group Vice President Jim Allchin was similarly vague. "It's all a question of probabilities," he said. "[2005 is] our target. But there's a probability it may make it, it may not ... The truth is, these are targets ... We'll know so much more when we hit Beta 1. And we're not going to be at Beta 1 at the PDC [Professional Developers Conference in late October]. Once we hit Beta 1, we'll be able to get customer feedback. You can't predict when a product is going to ship until you get some customer feedback."

Almost two years later, there was still no release date for the new operating system. What are the difficulties in estimating a release date for software?

Difficulties with Software Project Duration Estimation

1. One of the main problems associated with estimating software project duration is defining what is included in the product offering and what is excluded. As specifications are fleshed out and then agreed upon, there is a tendency for management to add to, or change requirements as customer feedback is received. This can clearly be seen from Jim Allchin’s comments in the quote above where he observes that customer feedback will alter the proposed schedule.

2. Boehm and Fairley mention the lack of clarity when defining what work is included in a software project’s definition. They say

‘Does an estimate of 100 person-months for “software development” include analysis and design, integration and test, deployment, management, or uncompensated overtime? If you use the estimate without knowing the answers [to these questions], you can get yourself into serious trouble’

Projects are often embarked upon without a clear understanding of exactly what any declared deadlines relate to.

3. Design documents are often not detailed enough, so that when it comes to the build phase of the project, there are too many unknowns still in existence. Tasks that were assumed to be straightforward turn out to be more complex and so take much longer to develop than was anticipated. In fact Armour describes the software developer’s role as

4. The level of maturity of the software development process in an organisation or team is another large factor. The more experience a team has in developing software, the more efficient it will become.

5. Another difficulty is the fact that most developers, when asked how long a task will take, tend to use very inaccurate ‘guestimations’. Developers are notoriously bad at estimating how long a task will take. There is a problem as Armour notes,

‘large commitments of lots of money are made by big companies based on hunches and guesses by junior programmers.’

6. Kolmogorov complexity, a well-established and accepted area of information theory and computer science, describes the inherent complexity of software. In effect it claims that there is no algorithm for finding the shortest program with a desired behaviour – or there is no best way of doing a project. When we accept that there is no best way of writing a piece of software, we realise that project length estimation cannot be done with 100% accuracy.

7. An application of the second law of thermodynamics is also relevant here. As there are far more things that will slow down a project than will speed it up, we almost always underestimate.
When one considers all these reasons why software project estimation is problematic the question must be asked, are there any techniques for maximising the accuracy of estimates. It is not good enough to dismiss it as an impossible task. Companies that invest in software development insist upon some sort of estimate so as to be able to plan budgets and schedules.

Current Techniques for estimating project length

The following are some techniques that are used by industry to improve estimates.

1. Guess

"Software is a manufacturing process and manufacturing isn't a guessing game. There is too much at stake," (Boz Elloy, senior vice president of software products at Borland Software). This is obviously the most inaccurate method of estimation. It uses the expectation that a developer or team of developers will have a good ‘feeling’ for the size and expected duration of a project based on their experience in past projects. One of the obvious difficulties is the amount of unknowns. Armour in his article ‘Ten Unmyths of Project Estimation’, stresses that a developer’s primary job is not to write code but to gain knowledge. He suggests that because ‘much of software development involves acquiring knowledge we do not already have, we can neither accurately estimate how long it will take us or even if we have got it all.’

A way of simplifying this guess is to break the work into smaller more manageable pieces and estimate how long each piece will take. This technique called modelling assumes however a good understanding of what the various elements of a project are. This is not always the case.

2. Improved Guessing – Modelling

Karl Wiegers once said, ‘If a picture is worth a thousand words, a model is worth 1024 lines of code’ (ouch!) Modelling is a very widely used technique that development teams use to get a better understanding of the various elements of a project and as a result can better estimate the amount of time needed to complete it. There are various techniques that are available for modelling including the Unified Modelling Language (UML). This is a standard that allows developers to describe the way parts of a program flow into each other and are connected. Although it does not focus primarily on the length of time each element will take to develop, it is the most popular model and gives a very good understanding of the overall product. Once these elements are identified, they can each be further broken down until the number of unknowns is reduced to a manageable size and better estimates can be made. According to Davis
‘A model simply provides us with a richer, higher level, and more semantically precise set of constructs than the underlying natural language. Using such a model reduces ambiguity, makes it easier to check for incompleteness and may at times improve understandibility.’

3. Improved Guessing – Benchmarking

This approach involves looking backwards in order to look forwards. When a team embarks on the process of estimating how long each element in a project will take, they will make many mistakes. Benchmarking involves looking back at how a software team has performed relative to its estimates in the past. This is the most accurate method mentioned thus far.

Software developers record what their estimates are prior to embarking on the project or task and then record the actual time it took. This has two main benefits. First it teaches developers how to estimate. Estimating is a skill that needs to be developed. By sitting down with a developer and reviewing his or her estimates, their accuracy can be increased over time.

The second benefit in recording estimates and actuals is that the success rate can be used in the future against future estimates. Furthermore, if a manager knows how long a certain task took in the past then a similar task should take a similar amount of time in the future. If a developer tends to be overly optimistic in his estimates by a factor of X then it is likely that he will do so again in the future. By recording the developer’s performance and success rate, future estimates can be more accurate.

A word of warning however. As each project is different, the length of time to perform the tasks in it will be different. Our ability to acquire knowledge we do not have however, tends to be similar, no matter what that knowledge might be.

Monte Carlo Simulation

Monte Carlo Simulation is a relatively new technique in the field of mathematics that uses simulation rather than precise analytical calculations to arrive at an estimated value. In essence it introduces noise into a mathematical analysis. In the real world, actual outcomes seldom match predicted outcomes that are based on probabilities. ‘Noise’ or unanticipated influences often encroach on a situation, causing spurious results.

Software development is a process where, as we have seen above, there are many factors that can influence the amount of time a project will take. As each project is different in terms of goals, environment and personnel (customers and team) the amount of ‘noise’ exerted on a project schedule is phenomenal.

The following is an application of Monte Carlo Simulation that may be used to allow the project manager better calculate a delivery date for a project. This is my own proposal that I believe is better for calculating an estimate as it factors in the fact that there are so many unknowns in software development. The steps are as follows: -

1. Get your team to estimate how long a task will take and then record the actual amount of time that task took. This gives us a series of estimates vs. actual for each team member.

2. Determine how far out each estimate is and express it as a percentage. For example, if an estimate was for 20 days and the actual was 30 days, then record that the estimate was 50% out.


3. Determine the frequency that each deviation occurred and calculate the probability that that deviation will occur again. Also work out a value between 0 and 99 that corresponds to that deviation. This value will be used later on.


4. For the next project that we are about to embark on, get Peter to estimate how long each task will take.

5. In this step we are going to inject some noise into Peter’s estimates. Noise by its nature is random so we get a series of random numbers from a random number generator.

6. We take each random number in turn and compare it with a value in the 4th column of step 3. This will give us an expected deviation and from that we can calculate the time differential.

For example, we take the first random number 39. By looking at the table in step 3 we see that 39 corresponds to a deviation of 21% -> 40%. We take Peter’s estimate of 9 days and combine it with the deviation and estimate that if this level of noise occurred, we would expect Peter to finish the task between 10.89 days and 12.6 days.

9 + (21% of 9 days) = 10.89 days
9 + (40% of 9 days) = 12.6 days

For each task we will add three different levels of noise and see what the best and worst case for Peter doing the task are.

We work out therefore that for Peter’s estimate of 9 days and given the level of accuracy he has shown in the past in his estimates, we should expect him to finish the task between 9.99 and 12.6 days. If we calculate out his other estimates we find the following

For the entire project we come up with the following estimates


Conclusion

While this method isn’t perfect, it moves beyond guessing and benchmarking to account of the noise that enters into all projects. Of course all the calculations and tables can be automated out so that its complexity is hidden from the user. I believe that it will give more accurate estimates however that organisations can use to improve their project estimations.

Random deviations will occur with the same frequency that they occurred in the past.

Enhance your TeamTrack Interface

In this tutorial I want to show you how you can use HTML to add pictures to your TeamTrack solutions in order to make them more interactive and colourful. A picture that uses shadowing can also add depth to your solution making it richer and more appealing.

The technique basically is made up of the following steps.

1. Make the graphic that you want to add and save it as a jpg file [e.g. ExplainImage.jpg] in the main images folder which is usually [C:\Program Files\Serena\TeamTrack\bin\images] on the server computer.
2. Add a new text field to the workflow called lblExplanationHTML. Set the type to 'Memo' and check the ‘Spans entire row on form’ checkbox. ‘Render HTML tags’ should also be checked
3. Check the ‘Access Overrideable Attributes’ box to make the ‘Default Value’ box editable and make it read only.
4. In the ‘Default Value’ box type the following
   
   

   <img src = "../tmtrack/images/ExplainImage.jpg">

5. Click the 'OK' button to save the new field
6. Edit the field again and replace the logical field name 'lblExplanationHTML' with a colon :If you add that field to any transition or state it will be visible as in the screen shot below

In a future article I'll show you how you can build complex colourful tables using HTML and scripting.

The science behind PPM

If you think about it, projects and investment funds have a lot in common. Both involve an initial outlay of costs but the hope is that they will bring a financial gain to the investor in the future. Given the constraints of finite resources, the investor will need to be clever about his choices. Selecting one project or investment fund will naturally exclude the selection of others. The selections cannot be based on a whim and must be approached in a dispassionate manner that allows each option to be quantified and mathematically compared.

Dr. Harry Markowitz won the Nobel Prize for economics in 1990 for his work on the Efficient Frontier theory. This theory looks at selecting a portfolio of investments so as to maximise the return to an organisation. Project and Portfolio Management (PPM) has its roots in investment fund portfolio management. As with financial investments, the idea is to get the maximum return for the mix of projects that a company can choose from.

This includes both selecting the right projects and periodically reviewing the selection to ensure that the mix continues to be the best possible for the organisation. This optimisation ensures that we neither overspend on IT nor miss revenue generating or cost saving opportunities for further investment.

The Efficient Frontier
The Efficient Frontier helps organisations understand the tradeoffs between portfolio value and cost. It effectively identifies the set of all portfolios that will give the highest expected return for each given level of risk. The organisation can then make a decision about the level of risk that it wants to adopt.

Obviously, to be able to compare portfolios in this way requires that information about projects and project execution be entered into a software application. Mariner is one such application that companies such as Starbucks and Yahoo have used to build successful business models.

Modelling as the future

This interesting blog article by Jeremy Burton looks at the focus Microsoft is putting on Modelling as the ‘Holy Grail’ of programming. The idea is that programmers will be able to develop applications in the future simply and quickly by linking together services using a graphical tool.

The developer essentially draws a picture of a process (model). When she is happy with that process she can turn it on and it becomes available to the entire organisation. The new process is automatically enforced which frees up employees from focusing on operational issues and instead allows them to spend their time pursuing strategic goals. This is already possible with Serena TeamTrack which organisations all over the world have been using for years to manage their businesses.

The next generation of modelling that Jeremy speaks of is the ability to have non technical people draw these pictures and be able to expand their process pictures to tie in with other applications in the organisation or websites that are outside it. These websites can include SalesForce, Google Earth, QuickBooks online, or any other website that exposes a web service.

While Microsoft talk about this Modelling as an aspiration, Serena is about to release a product that will accomplish it.

Business Mashups are coming early next year.