Software Estimation: Why It's Hard and How to Do It Better

The Estimation Problem Is Not What Most People Think

Every manager who has watched a software project run late has an opinion on why engineers can't estimate. The common theories: engineers are optimists who ignore risk, they don't account for meetings and interruptions, they underestimate complexity, they forget testing and code review.

All of these are partially true. None of them are the actual root cause.

The actual root cause: software estimation is inherently forecasting under uncertainty, and humans are systematically bad at forecasting under uncertainty — especially for novel, complex work. This isn't a character flaw. It's a cognitive limitation that applies to engineers, managers, and every other professional who estimates novel complex work for a living.

Understanding why estimation is hard is the prerequisite for doing it better.

Why Software Estimates Are Wrong

The Planning Fallacy

Nobel laureate Daniel Kahneman documented a systematic bias he called the planning fallacy: people consistently predict that their projects will proceed according to the best-case scenario while ignoring the base rate of similar past projects. When engineers estimate, they imagine how the task will go when everything works — no unexpected dependencies, no design dead-ends, no scope creep, no production incidents interrupting work.

This isn't wishful thinking consciously. It's the brain generating a narrative of task completion without adequately accounting for the class of things that might go wrong.

Unknown Unknowns

Estimation works reasonably well for work you've done before. The problem with software is that genuinely novel work — a new integration, an unfamiliar codebase, a problem you haven't encountered — has unknown unknowns. You don't know what you don't know until you encounter it.

Discovery work, exploratory design, and legacy system interaction are particularly estimation-resistant for this reason. Every time you think you understand the scope, you find another layer.

Requirements Drift

A task estimated as three days grows to ten days because the requirements evolved during development. This is often counted as an estimation failure when it's actually a requirements failure. The estimate was for a different scope than what was eventually built.

Optimistic Completion Rates

Engineers typically estimate how long a task takes when they're working on it — not accounting for meetings, context switching, code review cycles, deployment pipelines, and the reality that a "day of work" in most engineering organizations is three to five focused hours.

The Cone of Uncertainty

The cone of uncertainty is one of the most useful frameworks for communicating honest estimates. Developed by Barry Boehm and formalized in Steve McConnell's work on software estimation, it quantifies the range of possible outcomes at different stages of a project.

At project initiation, before detailed requirements are understood, an estimate might be off by a factor of 4x in either direction — what looks like a 6-month project might take anywhere from 1.5 to 24 months. As requirements are detailed and a high-level design is produced, the range narrows to roughly 1.5x in either direction. After a detailed design with specification of individual components, the range narrows further.

The cone of uncertainty is not pessimism. It's an accurate representation of what estimates mean at different stages. An estimate given before requirements are understood isn't really an estimate — it's an order-of-magnitude guess, and it should be communicated as such.

Communicating With the Cone

When someone asks for an estimate before the work is fully understood, give them a range, not a point estimate. "Based on similar work we've done, this is likely 3-6 weeks. Once we've done a spike to understand the integration complexity, I can give you a tighter range."

This communication is more honest, more useful, and more likely to result in good project planning than a falsely precise single number.

Reference Class Forecasting

Reference class forecasting is a technique developed by psychologists to counter the planning fallacy. Instead of estimating from the inside view (how this specific task will go), you estimate from the outside view (how tasks like this typically go).

The process:

Identify a reference class of similar past projects or tasks
Determine the historical distribution of outcomes for that class
Anchor your estimate on the historical distribution
Adjust (modestly) for features of this specific case that distinguish it from the class

In practice, this means keeping records. How long did similar past features take? What percentage of them came in on time? What was the average overrun factor? If your team's last five integration projects averaged 1.8x the initial estimate, your next integration estimate should probably be multiplied by 1.8.

Teams that track their estimation accuracy systematically — comparing estimates to actuals across a meaningful sample of work — are much better at estimating than teams that don't. The feedback loop is what calibrates intuition.

Estimation Techniques in Practice

Story Points and Relative Estimation

Agile teams often use story points to estimate relative complexity rather than time. A task assigned 3 points is roughly three times more complex than a 1-point task. Over time, the team's "velocity" (points completed per sprint) provides a throughput metric that can be used for capacity planning.

Story points sidestep the planning fallacy somewhat by focusing on complexity rather than duration. They work reasonably well for teams with stable membership and reasonably consistent work types. They break down for novel work types, team changes, and long-range planning.

Three-Point Estimation (PERT)

For individual tasks, three-point estimation provides a range: Best Case (B), Most Likely (M), and Worst Case (W). Expected duration is calculated as (B + 4M + W) / 6. This forces explicit consideration of the pessimistic case, which planning fallacy-prone estimators tend to ignore.

Decomposition

Break work into the smallest pieces you can before estimating. Large tasks are estimated poorly because they contain many unknown unknowns. Small, well-understood tasks are estimated better. If you can't break a task down below a week, that's a signal that the scope isn't understood well enough to estimate.

Time-Boxing Discovery Work

For genuinely novel or exploratory work, don't estimate it — time-box it. "We'll spend three days investigating the feasibility of this integration and come back with a better-scoped estimate." A time-boxed spike gives you the information you need to estimate the actual implementation without creating a false precision estimate upfront.

Estimation Anti-Patterns

Negotiating estimates. If a manager responds to an estimate by saying "that can't be right, we need it done in two weeks" and the engineer revises their estimate to two weeks, the estimate has become a commitment to deliver under the manager's preferred schedule. The underlying complexity hasn't changed. The outcome will be either low quality, burnout, or a missed commitment.

Padding without communicating. Engineers who've learned that estimates get negotiated down start padding them. This is rational but leads to inflated backlogs, poor prioritization, and the erosion of trust when padding is detected. The better approach: provide honest estimates with explicit uncertainty ranges.

Estimating without historical data. Making predictions without tracking outcomes produces no feedback loop. You have no way to know if your estimates are systematically optimistic, pessimistic, or inconsistent.

Ignoring task dependencies. Estimates for individual tasks don't account for sequencing, blocking dependencies, or critical path delays. Project timeline estimates need to model dependencies, not just sum task estimates.

What Good Estimation Practice Looks Like

Good estimation practice in a software team looks like:

Maintaining historical records of estimate vs actual for a meaningful sample of work
Communicating estimates as ranges with explicit confidence levels
Time-boxing discovery before estimating novel work
Decomposing before estimating
Treating requirements changes as scope changes that require estimate revision
Building in buffer for integration, testing, and code review — not just implementation

Estimation will never be a precise science. The goal is calibrated estimates — estimates whose uncertainty is accurate and whose track record is trustworthy. A team that consistently delivers within their estimated range, even if the range is wide, is doing something very valuable.

If you're working on improving estimation practices within an engineering team or building a planning process that's more honest about uncertainty, I'd be glad to connect.