In a previous blog post I highlighted Murphy's Law, which famously states "Everything that can go wrong, will". Today I will introduce you to Hofstadter's Law, another useful tool for programmers to keep in mind.
Hofstadter's law states:
It always takes longer than you expect, even when you take into account Hofstadter's Law.
This law ties into the Planning Fallacy proposed by Daniel Kahneman and Amos Tversky. The Planning Fallacy suggests that optimism bias impacts our predictions, resulting in underestimation of the time needed to complete a future task.
If you have been a developer for a moderate amount of time, you have discovered the difficult nature of estimating and scheduling software tasks. Our natural tendency is certainly tilted toward optimistic estimates. We make plans and commit to schedules based on our own best-case scenarios: nothing will go wrong, nothing will distract us, and we will fly through our work like savant developers. To make matters worse, managers view any attempt to add realism or failure contingencies to schedules as "sandbagging".
Our estimates can be difficult to create for a variety of reasons:
- Pressure to agree to specific milestones from investors, customers, or management
- Failing to focus our short-term goals
- Planning at a high level
- Lack of focus (e.g. multitasking and continual context switching)
- Failure to identify and complete the most important tasks (e.g. procrastination)
- Failure to account for common/potential delays:
- Urgent meetings
- Newly discovered problems
- Shifting priorities
- Upstream delays (e.g. in a manufacturing pipeline)
- Failure to acknowledge a limited supply of resources to accomplish the mission
In a world that focuses on immediate gratification, failing to meet a schedule produces extreme negative responses. I have watched developers and teams mutiny against schedules, estimates, and having project managers. If scheduling is unrealistic and sets false expectations, why plan anything?
Jerry Fitzpatrick, long-time consultant and author of The Timeless Laws of Software Development, offered these thoughts on scheduling and estimation difficulties:
Under-estimation is nearly universal, and not just in software development. Any team that assumes (or is pressured into) the best case scenario is being badly mismanaged. I've found that estimates are often wrong because we don't think deeply enough about the details and the potential side-effects of the task. In addition, far more time is spent on meetings, coffee breaks, planning and other activities than most people think.
I warn you not to take Hofstadter's Law as an indication that planning is pointless. Keep the law in mind so you can set expectations accordingly. You will be late - add some slop to your estimates, and if you're unsure about a task add even more time. Expect your project to be late even with this slop.
Jerry offers advice for those looking to improve their estimates:
An old adage is to multiply your initial estimate by three, but that really does amount to sandbagging. I like my "slop" to be more accurate by actually measuring the average percentage of my time used strictly for development. Then, for each task, I estimate the variance from nominal (also as a percentage). For example, I usually express an estimate this way: "22 hours +/- 20%". Though my estimate can still be wrong, it's almost always better than an off-the-cuff (under-) estimate.
Jack Ganssle, another long-time firmware consultant, recommends the Wideband Delphi technique for software estimation. The technique is nicely summarized in the paper Stop Promising Miracles by Karl Wiegers.
James Grenning also emphasizes the fact that our plans will be wrong. He proposes that we should find a faster way to plan, and then continually revise the plan as we learn. James's strategy involves relative difficulty rankings and is described in an article on his website.
I find that my ability to create estimates is greatly limited by the time horizon under consideration. When looking at timelines longer than three months, the chance of creating an accurate schedule seems impossible. There are always second- and third-order effects that we't taken into account. Chunk your projects into smaller sub-tasks, and you'll see improvements in your estimates.
Just remember: It always takes longer than you expect, even when you take into account Hofstadter's Law.
- <a data-preserve-html-node="true" href="https://en.wikipedia.org/wiki/Hofstadter%27s_law" target="_blank>Wikipedia: Hofstadter's Law
- Hofstadter's Law and Realistic Planning
- Murphy's Law
- The Timeless Laws of Software Development Planning Fallacy
- Wideband Delphi
- Stop Promising Miracles by Karl Wiegers
- The Planning Poker Party by James Grenning
- Added notes from James Grenning on estmiation
- Added comment from Jerry Fitzpatrick on under-estimation
- Added reference to Planning Fallacy per Ruth Malan's recommendation
- Added notes on improving schedule estimates
- Add more resources
- Cleaned up grammar and style