M08-9 Conducting A Cost Risk Analysis

8.0 - MANAGING COST ESTIMATING & BUDGETING

08.1 - Module 08-1 - Introduction to Managing Cost Estimating & Budgeting

08.2 - Module 08-2 - Develop Cost Estimating & Budgeting Policies & Procedures Manual

08.3 - Module 08-3 - Define The Estimates Purpose And Scope Of Work (Owner)

08.4 - Module 08-4 - Creating the Owners Cost Estimate (Top Down)

08.5 - Module 08-5 -  Define the Estimates Purpose and Interpret the Scope of Work (Contractor)

08.6 - Module 08-6 -  Developing the Contractors Cost Estimate (Bottom Up)

08.7 - Module 08-7 - Validate The Time & Cost Trade-Offs

08.8 - Module 08-8 - Validating Horizontal And Vertical Integration

08.9 - MODULE 08-9 - CONDUCTING A COST RISK ANALYSIS

Figure 1 - Conducting a Cost Risk Analaysis Process Map

Source: Guild of Project Controls

08.9.1 INTRODUCTION

This is the third and final “quality control check” on the cost and resource loaded schedule before we officially submit it for acceptance as the performance measurement baseline. This is the final check to see if the dates targeted by management internally or through the contract documents are achievable and to what degree (percentage) probability any given date has of being achieved, be it the overall project completion or any interim milestone date.

There are differing schools of thought on whether the project should be simulated WITHOUT contingency or buffers added at the activity level and they should be added AFTER the simulation is done. On the other hand, there is a strong argument that the schedule should be simulated as “designed by the people who are going to be doing the work” which means the schedule should be simulated exactly as those executing the plan have indicated.

It is the Guild’s position that lacking any compelling reason to do otherwise, that the schedule which is going to be used as the performance measurement baseline (PMB) against which to track progress, should be simulated exactly as it was designed and agreed to by those executing the work. This means contingency and buffers at whatever level are included in the simulation model.

Worth noting is the reason that simulation is the final of the three quality control checks is because if there are any errors in logic, duration or resource allocations AND/OR if the coding structures are not complete allowing for role up/roll down, then running the simulation will be less useful and in fact may produce totally invalid results. The GIGO Principle at work- Garbage In/Garbage Out.

08.9.2 INPUTS

Completely Cost And Resource Loaded Schedule Which Contains The Following:

• Work And Planning Packages For Each Wbs Element (100% Of Necessary Activities)
• No Missing, Redundant Or Incorrect Logic
• Milestones And Constraints
• Material, Equipment And Human Resources
• Activity Durations Which Are Realistic And Achievable
• Activity Costs (Activity Based Costing- Abc)
• Time And Cost Contingency Or Buffers Appropriate To The Riskiness Of The Project, Work Package And/Or Activity

08.9.3 TOOLS & TECHNIQUES

08.9.3.1 Dealing with Risks and Uncertainties

Experience in project scheduling shows that the probability of successful implementation of project schedules and budgets exactly to plan can be low. This risk is mitigated by well-designed and managed project controls processes, but the initial plan can carry a large risk of failure. A Risk Management Plan is often conceived and implemented to reduce this risk of failure.

As a result of uncertainty about the modelling of the means and methods, the estimate of project completion is just that, an estimate. It will be valid only to the extent that all the estimates of task or activity durations, assumptions about sequencing and resource availability, and production rates and project conditions all turn out to be true. The quality of the model has a strong effect on the likelihood of project success, however, even with the best models, there are multiple risks to anticipate and attempt to plan. The project completed is likely not to be as projected by the schedule (or put another way; by the deterministic CPM schedule) but is likely to be somewhere in a range of dates that like any random sampling, will be spread symmetrically around that estimated completion date.

Therefore project scheduling technology benefit from the inclusion of a risk simulation to produce probabilistic modelling of the projected probable completion dates. If these dates prove to be unacceptable then the schedule development process has to loop back to the beginning and a review done of the scope, the means and methods, the logic and the durations to see if the model as simulated can be done in the allotted time and if so, what is the probability of that happening.

Although this is clearly NOT a “best tested and proven” practice, the typical CPM schedule development often does not include risk assessments, but rather is developed as if all the components of the schedule, such as duration, are exact values, known by the planner. This is one of the reasons why so many projects are finishing late, and is a major ethical issue that the professional project control professional needs to face up to and work to change.These schedules include activity durations established by calculation of the quantity of work represented by an activity divided by the assumed or historical production rate. Risk management recognizes the uncertainty in planning and provides a system to brainstorm other risks that may occur during the project.

This brainstorming essentially identifies potential risks which are stored and managed during the life of the project within a Risk Register. (See Module 4- Managing Risk & Opportunity for an example of a Risk Register).

08.9.3.2 Duration Risk & Uncertainty

General duration uncertainty is the risk resulting from estimated durations (or costs as the two are correlated) being inaccurate or overly optimistic, based on assumptions not necessarily correct or accurate or based on the failure to conduct an adequate risk assessment.

When discussing duration uncertainty, many planners and schedulers or project members believe that their initial estimations of durations are accurate, and if the risk discussion continues, believe that their durations are the “most likely” durations. A problem with using such a “deterministic” approach is that many assumptions are made in early development of the schedule but rarely if ever questioned or reviewed again during the project.

So a “probabilistic” approach takes the concept of estimating durations based on average production rates to the next level; it looks at the range of possible project durations based on a spread (called “distribution” in statistics) in the estimated activity durations, commonly called three-point estimates of duration; the pessimistic, the most likely, and the optimistic.

The estimated durations, whether single estimates or three-point estimates, should be calculated by accurate quantities and appropriate production rates. This allows for a compiling of all general risks to the duration into this three point estimate by providing a range of durations that can be used in the risk analysis.

However, as we know from Module 4- Managing Risk & Opportunity using the “Average” or P50 value means that 50% of the time our activity will take longer than the duration entered and 50% of the time the activity will take less time. Assuming that the two will cancel one another out is a false assumption, evidenced by the number of projects which finish late.

This means that to be a responsible professional we need to increase the probability of finishing on time, which means we cannot use the “average” or P50 value but built in a time or money contingency to raise the probability from 50% to whatever probability or comfort level management wants. Usually for an owner organization this is P85% or P90% and for a contractor, that is normally P60% to P75%.

08.9.3.3 Monte Carlo Simulation

Using such a range of durations, the Monte Carlo technique makes the analysis of duration uncertainty and activity relationships risks possible. Monte Carlo analysis runs a large number of iterations based on the spread of the three duration estimates, so that many combinations of durations are used.

This probabilistic approach recognizes that the more accurate way to model uncertainty in durations is through the use of statistics, where if enough iterations are run, the results will generally fall into one of the common probability distributions of activity durations. One of those probability distributions that is commonly seen and used is the “normal distribution”, which graphs into a bell curve with the most likely duration at the highest peak of the curve and smaller probabilities as the curve diminishes in both directions.

Figure 2 - Example of Output from a Monte Carlo Simulation

Source: Source Courtesy of Hullett, David (2002)

In the example above, a simple project schedule was simulated.

(1) The target finish date established by management was 3 September

(2) The probability of finishing on/before that date is about 13%

(3) The “Most Likely” date of finishing is around 10 September which as a “probability” (P) of about 35% (P35)

(4) The probability of finishing on or before 13 September is 50% (P50) which means that the project has a 50% chance (P50) of finishing later. This is the DANGER of using “average” or “mean” durations in our schedules.

(5) The P85 (85% probability) of finishing on or before would be on 23 September.

(6) For those familiar with statistical analysis (See Module 4 - Managing Risk & Opportunity the Standard Deviation or Sigma (δ) is 8.98 or 9 days.

This is the whole purpose of testing the “as planned” schedule one last time before formally issuing it as a baseline schedule. IF we as project control professionals were to do this AND we were able to make a compelling example to the key decision makers as to why their project completion dates were unrealistic and assuming we can convince them to listen and heed our advice, then not only would more projects finish on time but there would a lot fewer claims and disputes on projects than we currently have.

The Monte Carlo analyses provide statistically significant “confidence levels" for the probabilistic prediction of achieving specific or targeted completion dates and - since schedules are dealing with unknowns - enable the schedules to have higher probabilities of meeting the chosen completion date. One weakness with Monte Carlo simulation is that it doesn’t take into account good project controls, where analysis and feedback from periodic updates is used to modify the plan. With the use of good project controls, it is much less likely that the stacking of pessimistic durations as happens in the simulations would go unnoticed and unmitigated. However, Monte Carlo simulation provides a number of very useful metrics as well as producing comparative evaluations of completion probability.

The Monte Carlo methods are available in numerous computerised software packages which can be linked to the scheduling software or are built into the software.

In addition, the level of detail in the schedule is an important requirement for use with risk analysis. There must be enough detail to isolate each activity by responsibility. This means that, for a schedule to be used in risk analysis, an activity should only model one discipline or party. This allows evaluation of each discipline or party for different assumptions of risk, as well as evaluation of areas for different risks. This is good scheduling or programming practice in any event, so it is in the best interests of the project for the schedule to contain this level of detail.

The use of constraints can affect the validity and accuracy of risk calculations because constraints may sequester float or prevent the network from calculating correctly. It may be necessary in the risk analysis to remove constraints in order to see the impact of those constraints on project completion. In addition, the use of mandatory constraints that do not allow the network to calculate float accurately in delay or gain is problematic and might render the schedule an inappropriate candidate for a risk analysis. Good practice in schedules suggests the minimum use of constraints, and avoidance of mandatory constraints that do not allow the network to calculate delay or gain, so following this practice will improve the results from Monte Carlo simulations as well as improve the quality of the schedule or programme.

Just as in any analysis of a schedule, the more calendars that exist in the schedule, the harder it is to analyze, whether for delay or risk. Multiple calendars with many shifts from calendar to calendar along critical paths will amplify or reduce the total float values. Good practice suggests minimum use of calendars, so following this practice will improve the results from Monte Carlo simulations as well as improve the quality of the schedule or programme. However, the project model should reflect the real situation and should use as many calendars as are necessary for accurate modelling.

The risk management plan is based on the schedule network calculations, whether using Monte Carlo analysis or what-if scenarios. So if the network does not calculate correctly, the value from risk management is severely reduced. This means that another of the generally accepted scheduling best practices, the limitation of open-ended activities to the first and last activities, is vital to risk management analysis.

With Monte Carlo analysis, the iterations are run with multiple values of activity durations, so those durations have to be reasonable, comparable, and ideally limited to one update period or less.

08.9.3.4 Network Logic Risk & Uncertainty

Network logic risks are those that generally occur as a result of scheduling decisions built into the network development related to the sequencing of activities determined by the activity relationships. The two most common risks in terms of network or logic are:

(1) Open ends or Hangars

See Figure 7 - Incomplete and Dangling Logic, in Module Module 7.4 - Create the Logical Relationships & Sequence Activities which is a risk event that can and should be fixed prior to the schedule being baselined and needs to be checked whenever activities are added to or deleted from the schedule after baselining.

This is the purpose of using Module 7.4 - Create the Logical Relationships & Sequence Activities followed by Module 7.8 - Validate The Critical Path & Completion Dates as “Quality Checks” on the schedule before it is simulated. If the logic is faulty, the durations or resources incorrect, then no amount of simulation is going to produce an accurate, reliable and precise schedule.

(2) Out of Sequence progress

This only happens after the schedule has been baselined and work has actually started. To see more on “Out of Sequence Progress” refer to Module 9.0 - Managing Project Progress.  If not fixed, “out of sequence progress” results in the earned value for the entire string of activities not being recorded.

08.9.3.5 Merge Bias or Merge Points

Merge bias is defined to be the multiplicative impact of having two or more parallel paths of activities, each with its own variability, logically linked into a single activity or milestone. If a number of paths originate or terminate in one activity, there is a significantly increased risk of delay should any one of these paths not be completed on time; these are sometimes referred to as Convergence or Merge Points or Path Convergence, “Choke Points” or “Bottlenecks”. This added risk at merge points occurs, potentially, at each merge point in the schedule. This is one reason why complex projects with many parallel paths and merge points are viewed as sources of high risk. This is why the Guild of Project Controls advises that unless absolutely necessary, logic links should be kept to less than three. For a better explanation of why this is true, see the example above, Figure ?? which gives us an example.

The logic of merge bias occurs, potentially, not just at the overall project finish milestone, but at any merge point in the schedule. For this reason particular attention should be given to these aspects in a project schedule.

Activities that require the same resource and have tight sequencing predictions are at a much higher risk of failure if resources are not as available as the schedule predicted, and failure in one activity will likely be reflected in all other activities that require that same resource.

The only way to see or measure the impact of merge bias or path convergence has on any activity is by using Monte Carlo Simulation.

08.9.3.6 Schedule Cost Contingency Approaches

There are two approaches to building either time or cost contingency into a schedule:

(1) We can apply Eli Goldratt’s Theory of Constraints and apply a BUFFER activity to the Project, Phase or even a string of Activities to counteract known (identified) risk events. The problem in using this approach is because everyone can see that it exists, the is a tendency for the work to expand to fill the allocated time, meaning it is not used for risk mitigation purposes but as “free” time for the resources to slack off or “dog” the project until the finish date. In the example shown, the mean (P50) value is 30 days and we added a 10 day long “buffer” activity. Knowing that the standard deviation (sigma) is 5 days, but adding the 10 day buffer the 30 day long activity, increases the probability of finishing this project, phase or activity within a 40 day period from 50% to 97%

(2) Below is exactly the same scenario but this time instead of breaking out the contingency so everyone can see it, we “embedded” or “buried” it so the only ones who know it is there are the people who put it there.

Which approach is “better”? It all depends on the circumstances but generally speaking, buffers that can be seen tend to get “used” while those which cannot be seen tend to be ignored.

For more on this, see Module 4- Managing Risk & Opportunity

Figure 3 - Two Approaches to building either time or cost contingency into a schedule

Source: Giammalvo, Paul D (2015) Course Materials. Contributed Under Creative Commons License BY v 4.0

08.9.3.7 Prioritizing Risks

Another powerful and useful tool/technique we can use total float for and that is to PRIORITIZE activities based on their total float which determines which activities are more IMPORTANT and the need date, which determines how URGENT they are.

This report can be created by rank ordering (sorting) by Early Start Date (classic waterfall) with a secondary sort by total float

In other words an activity which has negative total float will always be a top priority, as we know negative float is an impossible situation and we must make it go away. But assuming there is no negative float, then any activity with zero total float and has a scheduled start date of this week is a higher priority than an activity that has zero total float but isn’t due to start for another month.

Figure 4 - Urgency vs. Importance

Source: Giammalvo, Paul D (2015) Course Materials. Contributed Under Creative Commons License BY v 4.0

Obviously, if we discipline ourselves to follow this approach it should give us plenty of time to be able to identify those activities which represent the highest risk to the project and bring them to the attention of the project manager or other key decision makers BEFORE they become a crisis. Thus the priority would be:

(1) Negative Float whether urgent or not

(2) Urgent and Important (little or no float and coming up soon)

(3) Not Urgent but Important (little or no float so it may become negative if not attended to)

(4) Urgent but Not Important (approaching soon but some float to allow slippage)

(5) Not Urgent and Not Important. (has float and is off in the future somewhere)

Figure 5 - Calculating Expected Monetary Value (EMV) and Using it to Prioritize Risks

Source: Giammalvo, Paul D (2015) Course Materials. Contributed Under Creative Commons License BY v 4.0

The Risk Register, how to develop it and use it is covered in Module 4- Managing Risk & Opportunity. However for the purpose of PRIORITIZING RISK events it is done on the basis of Expected Monetary Value which is calculated by taking the probability of the risk even happening, times the impact which for scheduling is monetized by taking the total costs (direct and lost opportunity) for each day delayed and multiplying that times the probability. While this is a very common practice for cost estimators and project control professionals, planners and schedulers rarely seem to do this.

08.9.3.8 Probabilistic Branching

Probabilistic branching as shown in the GAO document represents only one example. The more likely future of scheduling utilizes systems dynamics software. Unlike the traditional forward pass/backwards pass software which does not allow for LOOPING, using Systems Dynamics software requires that those feedback loops be added.

This is a model which does not use the forward pass/backwards pass calculations. What this model does is views work as water flowing from buckets and flowing through a pipe. This pipe contains valves which modify the flow. Some of these valves are open when they should be closed and others are closed when they should be open. This is a way to model impact to the project by increasing (+) or decreasing (-) the flow rate of the water from one bucket to another through the pipes.

The most common use of systems dynamics that we see every day are the weather reports, more specifically, typhoons and hurricanes. Eventually, this is most likely going to be how projects are modeled.

For those younger practitioners, this is most likely the future of project management scheduling software and for those looking for a paper topic for the more advanced certification levels, or are looking for a Masters or PhD dissertation topic, this would be something you may want to look into.

Figure 6 - Probabalistic Branching Illustrated

Source: Lyneis, James and Ford, David (2011)  “System Dynamics applied to Project Management”



Figure 6-to the right shows the basic Systems Dynamics Model for Cost Plus type projects, while the graphic below shows the completed model for this same scenario, including those all important feedback loops which the current forward pass/backwards pass software packages do not allow.

While this example was created originally by Fluor for Cost Plus projects, there are also Systems Dynamics models created for traditional “Design>Bid>Build” (Firm Fixed Price) contracting.

Figure 7 - Showing what a Complete Construction Schedule looks like in Systems Dynamics software

ource: Lyneis, James and Ford, David (2011)  “System Dynamics applied to Project Management”

Notice in Figure 7 above that unlike the traditional software packages such as MS Project or Primavera, which use linear algorithm’s to calculate the early and late dates, systems dynamics creates a model of the project which yields probabilities of a project finishing on time and/or within budget.

For those interested in the evolving application of systems dynamics and systems thinking to project scheduling, here is a link to the paper where the above graphics are explained- “System dynamics applied to project management: a survey, assessment, and directions for future research” by James M. Lyneis and David N. Ford https://ceprofs.civil.tamu.edu/dford/dnf%20profesional/sdappliedtopm-sd…

And here is an additional link to a bibliography showing additional on-going research on this topic- https://ceprofs.civil.tamu.edu/dford/SDPMbibliography-110711a.pdf

The leading researchers on the topic of Systems Dynamics applied to scheduling in particular are:

• Sterman, John (1992) http://web.mit.edu/jsterman/www/SDG/project.html
• Love, Peter E.D. et al (2001) http://www.researchgate.net/profile/Peter_Love2/publication/223306343_U…
• Rodriquez, Alexander (1994) http://www.systemdynamics.org/conferences/1994/proceed/papers_vol_2/rod…
• Lyneis, James and Ford, David (2007) http://www.researchgate.net/publication/227643094_System_dynamics_appli…
• Ocap Zaynek (2012) http://iieom.org/ieom2012/pdfs/297.pdf
• Fluor Corporation (2012) https://www.youtube.com/watch?v=OeQcRjhhfcM
• Steve Warhoe (2014)
• https://www.linkedin.com/pulse/20140624160116-8896228-using-system-dyna…

08.9.4 OUTPUTS

• Updates To The Risk Register
• Probability Of Achieving Targeted Milestone Dates
• Risk Adjusted Durations (Tested And Validated Time Contingency, Tested And Validated Money Contingency)
• Risk Mitigating Buffers (Validated Time Buffers, Validated Monetary Buffers (Management Reserves)
• For Advanced Practitioners, Reference Also Best Practices Checklist: Conducting A Schedule Risk Analysis See Appendix I: An Auditor’s Key Questions And Documents Page 166-167
• For Advanced Practitioners Reference Also Best Practices Checklist: Conducting A Schedule Risk Analysis. See Appendix Iii: Standard Quantitative Measurements For Assessing Schedule Health Pages 188

08.9.5 REFERENCES & TEMPLATES

• The US Government Accountability Office (Gao) “Cost Estimating And Assessment Guide- Best Practices For Developing And Managing Capital Program Costs” (2009) Gao-09-3sp Http://Www.Gao.Gov/New.Items/D093sp.Pdf  
• USDept Of Energy (Doe) “Doe G 413.3-21, Cost Estimating Guide” Https://Www.Directives.Doe.Gov/Directives-Documents/400-Series/0413.3-Eguide-21/View
• Gary Cokins And The Institute Of Management Accounts (2006) “Implementing Activity Based Costing” Http://Www.Garycokins.Com/Images/Pdfs/Cokins%20ima%20sma%20implementing%20abc.Pdf
• United States Department Of The Interior National Park Service Class A Construction Cost Estimate Basis Of Estimate- Http://Www.Nps.Gov/Dscw/Upload/Classaconstcostestimatesample_1-26-11.Pdf )

08.10 - Module 08-10 - Baselining And Communicating The Cost Estimate/Cost Budget

GPCCAR M08-9, Revision 1.01