(originally written on 1/27/2014 for Prof. Fox’s class assignment, revision 1/22/2022)
As a software project manager, one of the job responsibilities is to estimate the effort and time for the project delivery, typically before the project starts. Management uses this information to estimate the cost and plan for the roadmap of the product. However, estimation accuracy has always been a problem in many of my past experiences. Cognitive biases have influenced estimation accuracy quite significantly and a manager needs to have awareness.
The Estimation Process
The approach we commonly use for the estimation is Planning Poker, a structural and commonly accepted method in the industry.
- The business requirements are broken down into smaller pieces, or “stories”.
- A group of 3–4 developers, including the tech lead, sit in a room with the project manager and business requirement analyst to discuss and estimate each story.
- Every developer receives a set of “t-shirt size” cards that has “S”, “M”, “L”, or “XL” letters on them (sometimes Fibonacci numbers).
- The analyst presents each story and all developers think about it and ask questions.
- Once all the questions are cleared, the project manager will ask every developer to present their cards at the same time to get an estimate.
- If they vary, they will discuss and reconcile the difference to an agreed letter/number. Each story may take an average of 2 to 5 minutes to estimate depending on its complexity.
- The end result of this estimation exercise is a list of relative and comparable t-shirt size letters on each story.
After all the estimates are done, the project manager will ask the developers to estimate a letter-to-dev-days conversion ratio so that we can relate the t-shirt sizes to the real person-days for planning purposes. The whole estimation process happens in a day or two to get a fast forecast of the project, as digging down into details will require significant efforts of the already busy team.
This approach considers some degree of the psychological aspects of human behaviors. For example, it takes the group decision principles into account. Instead of only asking the tech lead for a guess, it seeks input from multiple developers to make better group decisions. The estimation process is essentially a judgemental process of future probabilities, so the “consistency” principle applies in order to make a better outcome. It is best if developers make consistent and relative comparisons among the stories instead of using absolute days/hours on each story.
Cognitive Biases Impact Estimation
However, there are a few issues with this approach that could lead to a biased outcome. To start with, as there is always a time constraint, to get a result in a short time frame, it somewhat forces the developers to use their intuition to make the estimation other than to use their system 2 thinking (Kahneman, 2011). Prior to walking to a meeting room, the developers usually do not have any knowledge of the details of the business requirements they were about to estimate. Although they can ask questions, in 2 to 5 minutes it is hard to gain an in-depth understanding, and they have to make decisions based on what the analyst presented to them. If there is a push to get smaller estimates, which is usually expected by the project stakeholders, the team may take the expectation as an anchor and create underestimated results.
The availability biases and especially biases of imaginability (Tversky & Kahneman, 1974) could impact people’s ability to estimate. The estimation process will predict the risks and feasibilities of the future project. Developers base on both technical knowledge and their past experience to make the prediction. However, normal humans cannot accurately remember all details of past situations. The most vivid memory and most available information will be naturally recalled. People are programmed to call to mind easy-to-imagine risks and often overlook hard-to-imagine risks.
Poorly executed group decision also impacts the estimation process. The group polarization effect could happen when an influential person is in the room during the estimation. For example, the tech lead is usually the most senior person. Other developers who have a much different perspective but junior may just give up their idea due to authority bias.
Furthermore, the positive illusion could impact the team’s estimates (Kahneman, 2011, ch 19). People tend to be optimistic about the challenges and overconfident about their skills. Although via the user stories we can break down a large project into smaller pieces, when estimating each piece, people tend to neglect the other factors during the project execution, e.g. testing effort, integration, dependencies, communication, and collaboration with other teams. They tend to overestimate their ability to complete plans in a timely manner due to overconfidence.
Estimating the conversion ratio between the letter size and the actual days is the vaguest and most inaccurate part. The current 4 letters scale has introduced an anchoring effect (Tversky & Kahneman, 1974), where for some stories its size may be an XXXL but just because there is no such a category, the team will set it to the XL as it anchors on this range.
The conversion ratio will apply to the whole team and project, ignoring that they may be changing over time. The developers estimating the project use their self-confidence as the belief of the team’s average skills level, which is also usually not true. The estimators are generally more experienced as we selected experienced developers to estimate. Using themselves as anchors will overestimate the team’s average ability.
In this step, relying on historical data may introduce another anchor to the prediction because the new project is different. It could be much different from the previous ones or be riskier.
Approaches To Reduce The Biases
Simple Approaches
The estimation process has a built-in philosophy, “Consistency” and “Comparable” principles, to reduce the biases. To ensure them, at the end of the estimation, the project manager should list all estimated stories and group them by their estimated categories. They can ask the team to do a sanity check to evaluate if all the stories within the same “estimated bucket” are roughly the same size and have similar risks. Outliers should be discussed and reestimated.
Another simple approach a lot of managers use is padding. Sometimes adding 20% to 50% buffer to the estimated result is a wise choice to not go over budget and schedule. In Microsoft Project, adding a project buffer will simply add padding on top of the project schedule. Nowadays, many companies discourage managers to add any buffer as every dollar you didn’t spend at the end is a value trade-off from other teams’ projects.
Linear Regression Bootstrapping
To improve the estimation accuracy, a linear regression bootstrapping approach could be applied. Developers have some criteria in mind when they estimated the sizes of stories. We can reveal these criteria as variables and use them to find the predicted values based on a regression model.
For example, a team can rate the following aspects as part of their estimation factors:
- development effort
- whether the story involves third-party integration
- testing complexity
- communication
- collaboration needs with another team
- and the team’s familiarity with the technology.
These factors can be evaluated and rated by numeric values. The estimated result (either a Fibonacci number or a T-shirt size category) could be seen as the dependent variable. Then we can build a multiple linear regression model to find the expected estimates of each story. This is called bootstrapping (Fox, 2014, session 2) and it normalized the parameters and their correlations and reduced the inconsistency of personal estimates.
Monte Carlo Simulation
Monte Carlo simulation is another effective approach to estimating uncertainties and reducing biases. (I will write details in another article). Monte Carlo simulates the estimation with random sampling from the historical estimates, and with the large sample size, the estimated size of the project will form a bell curve. Then the managers can choose a confidence level to manage or accept the risk.
Even with advanced approaches like the improved bootstrapping approach and Monte Carlo simulations, however, we should admit that we couldn’t completely eliminate inaccuracies. After all, our future always has uncertainties and can not be predictable at all.
REFERENCE
- Fox, C. (2014). Managerial Decision Making, Anderson School of Business
- Kahneman, D. (2011). Thinking, Fast & Slow (TF&S)
- Lovallo, D. & Kahneman, D. (2003) Delusions of Success: How Optimism Undermines Executives’ Decisions. Harvard Business Review, 56–63
- Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. science, 185(4157), 1124–1131.
