Friday, November 16, 2007

Using Monte Carlo Simulation to derive more accurate project duration estimates

Monte Carlo Simulation is a relatively new technique in the field of mathematics that uses simulation rather than precise analytical calculations to arrive at an estimated value. In essence it introduces noise into a mathematical analysis. In the real world, actual outcomes seldom match predicted outcomes that are based on probabilities. ‘Noise’ or unanticipated influences often encroach on a situation, causing spurious results.

In this article I am going to look at how Monte Carlo Simulation could be used to give more accurate project duration estimates.

No project runs entirely uninterrupted from start to finish. Team members become ill or other higher priority tasks arise and require immediate attention. This is the ‘noise’ that I am referring to. ‘Noise’ is basically all the things that interfere with the normal running of a project. By looking back at the accuracy of a team’s estimates in the past and injecting random noise, the hope is that a more accurate estimate can be attained.

Forecasting how long a project is going to take has been a near impossible task for companies for years. Few large infrastructure projects where there is a team and strict deadlines in place, are completed on time.

Hugh W. Ryan, in an article for Outlook Journal, summarized research that showed:

- Only 8 percent of applications projects costing between $6 million and $10 million succeed.
- Among all IT development projects, only 16% are delivered to acceptable cost, time and quality.
- Cost overruns for IT projects have been estimated at $59 billion in the United States alone.

According to the Economist, a £456m ($844m) project for Britain's Child Support Agency came in over a year late, and has failed to deliver payments to more than half of eligible applicants and according to Cuppan approximately 80% of the medium to large software projects executed within the Department of Defense (DoD) were 100% over budget and 90% were at least one year behind schedule.

As the main cost in software development is developers, time over-runs quickly translate into cost over-runs. The accurate calculation of how long a project is going to take is vital to the success of a project. With a team of 20 developers, for example, that is costing €30,000 to €40,000 per week, delays soon add up.

In looking for an example of a large project that has run way over original estimation, consider the repeated delays of Microsoft’s Vista operating system. Vista was intended to replace Windows XP as Microsoft’s main operating system. While it is ambitious in terms of what it hoped to achieve (new file system structures and interface standards), it has become something of a joke in the industry. Every few months, Microsoft came out with a new vague release date intention which was ultimately not met.

When asked about Vista during an interview in Sept 2003, Microsoft Group Vice President Jim Allchin was similarly vague. "It's all a question of probabilities," he said. "[2005 is] our target. But there's a probability it may make it, it may not ... The truth is, these are targets ... We'll know so much more when we hit Beta 1. And we're not going to be at Beta 1 at the PDC [Professional Developers Conference in late October]. Once we hit Beta 1, we'll be able to get customer feedback. You can't predict when a product is going to ship until you get some customer feedback."

Almost two years later, there was still no release date for the new operating system. What are the difficulties in estimating a release date for software?

Difficulties with Software Project Duration Estimation

1. One of the main problems associated with estimating software project duration is defining what is included in the product offering and what is excluded. As specifications are fleshed out and then agreed upon, there is a tendency for management to add to, or change requirements as customer feedback is received. This can clearly be seen from Jim Allchin’s comments in the quote above where he observes that customer feedback will alter the proposed schedule.

2. Boehm and Fairley mention the lack of clarity when defining what work is included in a software project’s definition. They say

‘Does an estimate of 100 person-months for “software development” include analysis and design, integration and test, deployment, management, or uncompensated overtime? If you use the estimate without knowing the answers [to these questions], you can get yourself into serious trouble’

Projects are often embarked upon without a clear understanding of exactly what any declared deadlines relate to.

3. Design documents are often not detailed enough, so that when it comes to the build phase of the project, there are too many unknowns still in existence. Tasks that were assumed to be straightforward turn out to be more complex and so take much longer to develop than was anticipated. In fact Armour describes the software developer’s role as

4. The level of maturity of the software development process in an organisation or team is another large factor. The more experience a team has in developing software, the more efficient it will become.

5. Another difficulty is the fact that most developers, when asked how long a task will take, tend to use very inaccurate ‘guestimations’. Developers are notoriously bad at estimating how long a task will take. There is a problem as Armour notes,

‘large commitments of lots of money are made by big companies based on hunches and guesses by junior programmers.’

6. Kolmogorov complexity, a well-established and accepted area of information theory and computer science, describes the inherent complexity of software. In effect it claims that there is no algorithm for finding the shortest program with a desired behaviour – or there is no best way of doing a project. When we accept that there is no best way of writing a piece of software, we realise that project length estimation cannot be done with 100% accuracy.

7. An application of the second law of thermodynamics is also relevant here. As there are far more things that will slow down a project than will speed it up, we almost always underestimate.
When one considers all these reasons why software project estimation is problematic the question must be asked, are there any techniques for maximising the accuracy of estimates. It is not good enough to dismiss it as an impossible task. Companies that invest in software development insist upon some sort of estimate so as to be able to plan budgets and schedules.

Current Techniques for estimating project length

The following are some techniques that are used by industry to improve estimates.

1. Guess

"Software is a manufacturing process and manufacturing isn't a guessing game. There is too much at stake," (Boz Elloy, senior vice president of software products at Borland Software). This is obviously the most inaccurate method of estimation. It uses the expectation that a developer or team of developers will have a good ‘feeling’ for the size and expected duration of a project based on their experience in past projects. One of the obvious difficulties is the amount of unknowns. Armour in his article ‘Ten Unmyths of Project Estimation’, stresses that a developer’s primary job is not to write code but to gain knowledge. He suggests that because ‘much of software development involves acquiring knowledge we do not already have, we can neither accurately estimate how long it will take us or even if we have got it all.’

A way of simplifying this guess is to break the work into smaller more manageable pieces and estimate how long each piece will take. This technique called modelling assumes however a good understanding of what the various elements of a project are. This is not always the case.

2. Improved Guessing – Modelling

Karl Wiegers once said, ‘If a picture is worth a thousand words, a model is worth 1024 lines of code’ (ouch!) Modelling is a very widely used technique that development teams use to get a better understanding of the various elements of a project and as a result can better estimate the amount of time needed to complete it. There are various techniques that are available for modelling including the Unified Modelling Language (UML). This is a standard that allows developers to describe the way parts of a program flow into each other and are connected. Although it does not focus primarily on the length of time each element will take to develop, it is the most popular model and gives a very good understanding of the overall product. Once these elements are identified, they can each be further broken down until the number of unknowns is reduced to a manageable size and better estimates can be made. According to Davis
‘A model simply provides us with a richer, higher level, and more semantically precise set of constructs than the underlying natural language. Using such a model reduces ambiguity, makes it easier to check for incompleteness and may at times improve understandibility.’

3. Improved Guessing – Benchmarking

This approach involves looking backwards in order to look forwards. When a team embarks on the process of estimating how long each element in a project will take, they will make many mistakes. Benchmarking involves looking back at how a software team has performed relative to its estimates in the past. This is the most accurate method mentioned thus far.

Software developers record what their estimates are prior to embarking on the project or task and then record the actual time it took. This has two main benefits. First it teaches developers how to estimate. Estimating is a skill that needs to be developed. By sitting down with a developer and reviewing his or her estimates, their accuracy can be increased over time.

The second benefit in recording estimates and actuals is that the success rate can be used in the future against future estimates. Furthermore, if a manager knows how long a certain task took in the past then a similar task should take a similar amount of time in the future. If a developer tends to be overly optimistic in his estimates by a factor of X then it is likely that he will do so again in the future. By recording the developer’s performance and success rate, future estimates can be more accurate.

A word of warning however. As each project is different, the length of time to perform the tasks in it will be different. Our ability to acquire knowledge we do not have however, tends to be similar, no matter what that knowledge might be.

Monte Carlo Simulation

Monte Carlo Simulation is a relatively new technique in the field of mathematics that uses simulation rather than precise analytical calculations to arrive at an estimated value. In essence it introduces noise into a mathematical analysis. In the real world, actual outcomes seldom match predicted outcomes that are based on probabilities. ‘Noise’ or unanticipated influences often encroach on a situation, causing spurious results.

Software development is a process where, as we have seen above, there are many factors that can influence the amount of time a project will take. As each project is different in terms of goals, environment and personnel (customers and team) the amount of ‘noise’ exerted on a project schedule is phenomenal.

The following is an application of Monte Carlo Simulation that may be used to allow the project manager better calculate a delivery date for a project. This is my own proposal that I believe is better for calculating an estimate as it factors in the fact that there are so many unknowns in software development. The steps are as follows: -

1. Get your team to estimate how long a task will take and then record the actual amount of time that task took. This gives us a series of estimates vs. actual for each team member.

2. Determine how far out each estimate is and express it as a percentage. For example, if an estimate was for 20 days and the actual was 30 days, then record that the estimate was 50% out.

3. Determine the frequency that each deviation occurred and calculate the probability that that deviation will occur again. Also work out a value between 0 and 99 that corresponds to that deviation. This value will be used later on.

4. For the next project that we are about to embark on, get Peter to estimate how long each task will take.

5. In this step we are going to inject some noise into Peter’s estimates. Noise by its nature is random so we get a series of random numbers from a random number generator.

6. We take each random number in turn and compare it with a value in the 4th column of step 3. This will give us an expected deviation and from that we can calculate the time differential.

For example, we take the first random number 39. By looking at the table in step 3 we see that 39 corresponds to a deviation of 21% -> 40%. We take Peter’s estimate of 9 days and combine it with the deviation and estimate that if this level of noise occurred, we would expect Peter to finish the task between 10.89 days and 12.6 days.

9 + (21% of 9 days) = 10.89 days
9 + (40% of 9 days) = 12.6 days

For each task we will add three different levels of noise and see what the best and worst case for Peter doing the task are.

We work out therefore that for Peter’s estimate of 9 days and given the level of accuracy he has shown in the past in his estimates, we should expect him to finish the task between 9.99 and 12.6 days. If we calculate out his other estimates we find the following

For the entire project we come up with the following estimates


While this method isn’t perfect, it moves beyond guessing and benchmarking to account of the noise that enters into all projects. Of course all the calculations and tables can be automated out so that its complexity is hidden from the user. I believe that it will give more accurate estimates however that organisations can use to improve their project estimations.

Random deviations will occur with the same frequency that they occurred in the past.

No comments: