A simple way to improve your estimating (and a cool pub trick) – Conclusion
…and we’re back!
Well… that was a long commercial break wasn’t it 🙂
In case you missed part 1 of our version of the show “deal or no deal”, you missed the big cliff-hanger and you really should read part 1 first. For the rest of you, to quickly recap, I came out of the closet and admitted by secret teenybopper shame, told the world that my wife had a teenage thing for Jean Claude Van Damme, showed the effect of beer goggles and introduced the notion of cognitive bias and how it can affect judgement.
i also demonstrated how, by altering the frame of reference, to a problem something that at first seems completely unquantifiable “how the hell do I know how many SharePoint developers drive yellow cars?”, is actually not as “impossible” as you may first think.
At the end of the last post I left you with a $10000 dilemma. You had to make a “deal or no deal” decision about going with your estimate about SharePoint developers who own yellow cars, or to instead cast your lot with a bag of marbles with a 9 in 10 chance of winning the prize. Just to refresh your memory, here is the salient part of the pub conversation.
- Me: Okay, so you are 90% sure that here are between 300 and 2000 SharePoint developers in the world with a yellow car?
- Them: Yes
- Me: So, let’s make this like the game show “deal or no deal”. If you are right and the answer is within your range, you will win $10000. BUT you have an alternative…
- Them: Ok…
- Me: What if I were to present you with a bag containing 9 red marbles and 1 black marble and offer you $10000 if you pull out a red marble. Pull the one black marble, and you miss out on the money. Do you want to stick to your estimate or do you want to draw a marble?
So have you decided? Now be honest and see how you went against the 4 outcomes that I have experienced when trying this on people. Here are the possible answers in order of popularity…
- The person chooses to pull from the bag of marbles rather than their ranged estimate. (This is the predominant answer for all people I have tried this with – perhaps 70-75% of all responses).
- The person chooses to use their estimate over the bag of marbles. (perhaps 25% of people have answered with this option)
- Upon hearing the bag option, the person wants to change their ranged estimate. (Happened to me once)
- The person doesn’t care which method.. (never happened to me)
So which is the right answer to this question?
(drumroll) Lets tackle the possible answer in order of likelihood.
“Take the marble! take the maaaaaarble!”
For the 70 odd percent of you who opted to take your chances with the bag of marbles, GONG! you lose!
Better double check your estimates in future because you have demonstrated that you are over-confident in your estimates. In other words, you are suffering from optimism bias. To explain why, think about the original question carefully. I asked originally for a ranged estimate that you were 90% confident with.
I then presented an alternative that has a 9 out of 10 chance of success – also 90%. From a statistical point of view, you should be completely ambivalent as to which option to use. Therefore, despite being asked for a range that you were 90% confident with, the range you actually estimated is not really 90% at all. It has to be less than 90% for you to prefer a clear 9/10 probability.
So that is why you are so stressed and busy! You keep giving crap estimates that make life harder for you! 🙂 Either that or you are too nice and when your project manager looks at you with those big, sad project manager eyes, your heart melts and you relent.
Isn’t that cool in a nerdy way? It is very interesting to see people’s faces as the penny drops to this logic and they suddenly realise just how bad some of their past estimates have been as a result. The consolation prize is just about 4 out of 5 people do exactly the same as you and take the marbles.
“No deal, I will stick with my estimate”
For the smaller group who decide that their estimate is preferred, you also lose.
In this case, the reason why should be pretty obvious. You are so paranoid about getting it wrong, that you have made an estimate that is more like 95% or even 99% confident. Why? your range is too wide for 90% because when presented with a clear 9/10 chance of success, you chose your original estimate. While that may sound like you are confident, in reality you are a bit of a wuss, because in fact you are under confident with your estimate. So grow some balls you weenie 🙂
Honorary mention – “I want to change my estimate”
At the Best Practice Conference in DC, I attempted this pub trick on Yoav Lurie from Synteractive, who is much more of a business and strategic thinker than us IT nerds. His response I think, deserves an honorary mention for being the closest to winning the game. In this example, I asked him to estimate in feet, the wingspan of a Boeing 747. I knew instantly that he was a good estimator because of the logic he used to come to a range.
“Hmm, well an aircraft seat is maybe one and a half feet, and there will be 10 seats in the cabin, with two passages that are probably two feet in width…so that ads up to…”
What do you notice about what Yoav did? Straight away, he related the wingspan of an aircraft (a clear unknown), to something he could make a reasonable estimate of (the width of an aircraft seat). After all, we have all sat in an aircraft seat in sardine (economy) class and know how cramped it is. He knew there were three rows of seats and related this to the width of the cabin, which he then related to the size of the wing. Deducing that the wing might be 4 to 6 times the width of the cabin, he then was able to make a very good ranged estimate of the overall wingspan of the plane.
I was very impressed at his estimate and how he arrived at it, but I still got him 🙂
As soon as I presented him with the bag of marbles alternative, without missing a beat he said “I want to change my estimate”. It took only a split second of presenting a clear 90% probability made Yoav realise that his estimate was not 90% and he was still a little overconfident.
That being said, Yoav’s method of relating something known to help frame the reference to something unknown is the only time anyone has used any sort of rigour in forming an estimate and very impressive for the pub setting 🙂
The right answer
Okay, so as you may have guessed by now, the right answer is to shrug your shoulders and say “I don’t care” or wave your hand at me and say “pfft, whatever”. (This is one of the few times saying you couldn’t care less is the right answer). In doing so, you have placed equal weight upon the choices, based on the assumption that both are 90% probabilities.
Neat pub trick huh? It certainly gets people thinking.
How to calibrate yourself
Douglas Hubbard talks about “calibrated estimates” in his books and has an appendix of calibration questions, that are designed to help you perceive and account for cognitive bias in your estimating.
What you should take away from this exercise is that when asked to estimate on something you are uncertain about, make your initial estimate. Then, pretend you are in the game show and you have to pick between this estimate and the marble. If you feel that you would take the marble over your estimate, increase the width of your range until you feel that it doesn’t matter which option you pick.
Conversely, if you are one of the wimps who are under confident, then reduce the width of your range, until you feel that you have no particular preference of your estimate vs. the marbles.
In the same way that reframing a problem led from something being unquantifiable to something that indeed had a upper and lower range, by reframing the estimate against a unambiguous probability such as a bag of 10 marbles with 9 red, helps you to account for cognitive bias in your estimates.
Conclusion
So to reiterate my key points to this post
- Many things that seem unquantifiable are easier to quantify than you think, once you think in terms of ranged estimates and probability.
- Your bad taste in fashion and music when you were a teenager still manifests itself today and it is called cognitive bias.
- There are easy methods that you can use to calibrate yourself better so that your estimating radar is more finely tuned.
Most importantly of all however, you learned that my wife liked Jean Claude Van Damme in the 80’s and you know that I am in big trouble when she reads this! 😛
Thanks for reading
Paul
I would probably take the marble (unless my estimated range was huge). With the marble, the chances of winning are absolutely defined. However, with range confidence, there may not even be a meaningful metric – just a “gut feeling.”
Let’s say I’ve made my best effort at calculating the number of Sharepoint devs who drive yellow cars (or, more interestingly, how long it will take to finish a given story). That’s one calculation, necessarily imprecise. I attempt to quantify that imprecision (or “lack of confidence”) by making a different (but related) calculation. That calculation could be based, for instance, on how right (or how close to right) I’ve been in the past on similar questions
The point is that the first calculation (the range of values for the answer to the question) and the second calculation (the probability that the first calculation is correct) are both imprecise and based on imcomplete information. If your range was 0-infinity, then, yes, you could give it a 100% confidence. If the range is infinity-0, the confidence is would clearly be 0%. But for any realistic estimate of a) the range, and b) the confidence that the range was calculated correctly, you are dealing with far more complicated matters than the number of marbles in a jar, hence you have a far greater chance of being mistaken.
I still think the technique you write about is very interesting and valuable, however.
Hubbard back-tests this technique by giving you sample questions and scenarios pre calibration and post. Pre calibration, the success rate of participants who are 90% confident are more like 50 or 60%. Clearly a gap between the probability that people assign versus the results. He maintains that post calibration, the gap between stated probabilities and actual results narrows to a much smaller range.
As a result, estimates are much improved and the negative effect of optimism bias is reduced significantly.
thanks for the reply
Paul
Very interesting exercise.
Yet, I’m a bit surprised by your explanation. If I take the Marble, doesn’t it mean that I am not confident that my guess is 90% accurate? Then I would be under confident, not over-confident with myself as you suggested. Did I get it wrong somewhere?
No, if you take the marble, then your estimate must be less than 90% in reality, yet you made that estimate when being asked to be 90$ confident. Therefore, your original estimate was overconfident… make sense? 🙂
As you explain it, it kind of makes sense. Yet if I choose to take the marble, it means I am aware that my estimate was less than 90% accurate. Would you take it as meaning that I mean aware of my over-confidence ?
Yes I would take it in the way you describe… and therefore as a PM I would not have much faith in your estimate 😉
If you are aware of your overconfidence, you should change your estimate to a wider range until you feel that the marble *feels* the same as your range. In doing so, you are self calibrating and accounting for your optimism bias.
As I PM I might whine about the width of the range, but at the end of the day I shouldn’t force or coerce you to be more confident unless I provide you with more time to think about it, or more information to help you reduce your ranged estimate to a range that I can work with. To do anything less is to perpetuate the problem.
Asking for an estimate and then saying something like “that estimate is not good enough – halve it” is a pretty big warning sign in my book.
regards
Paul
Not a bad theory, I’ll give it a ride. I do however like the first guys point of knowing that the marbles is 90% versus feeling your estimate is 90%. I’d take the marbles.
This cognitive bias stuff sounds interesting to me, it’ll give me something to look into on a lazy friday.
Hey, just in case you’re looking for something to do, you could write more on the estimation process if you want as it’s something that my group struggles with on a regular basis. I come from a perspective of “under promise, over deliver” so my estimates often come in high. It’s usually tougher for the customer to swallow up front but it protects me in the end. We also run into problems where we have to do estimates based on sales calls/information where we don’t even get to talk to the customer first hand. Also, the person doing the estimate is often not the individual that’s doing the implementing so there may be a gap in skill sets. We try and build for a certain amount of scope creep as we feel it gives us leverage and we all know it happens. It seems to be a crap shoot on projects that come in under or on time. So how much of the problem is the estimate and how much is controlling the project?
Either way, good stuff as always.
Hey JT
I have made a habit of answering such questions in a range. I prefer being vaguely right than precisely wrong. Then when the ineviable complaint comes that the range is too wide, I then argue that a workshop will help me reduce my uncertainy. Hubbard takes this considerably further than me with his concept of the value of perfect information stuff where he quantifies whether the amount of effort required it worth the gain of reduced uncertainy. Most of my clients are happy to spend a couple of extra hours to reduce uncertainty significantly. It is amazing how face to face dialogue can do this.
From my point of view, the simple logic of the 90% test is sufficient to warrant further investigation. A client of mine recently said to me “Confidence is the feeling you have before you really understand the problem” and I think that is a particularly insightful quote.
Interestingly, the argument that the person doing the estimating is not the persion doing the work violates a lot of that I stand for principalwise, and dialogue mapping is my tool of choice for dealing with this (check this article for more on this angle http://www.cleverworkarounds.com/2009/09/18/am-i-a-business-analyst-what-about-those-calling-themselves-bas/).
At the end of the day, if this methods causes people to stop and reconsider an estimate, then it has had its intended effect.
regards
Paul
Interesting approach Paul. So if you’re comparing your own estimates against actuals, have you yourself improved as an estimator, ruling out increased experience?
Hi Michael
The experience thing is simply trusting yourself and resisting coercion into making your estimate fit nicer into a project plan. If you are asked to do this, then the plan is flawed, assuming you are calibrated. People often forget what estimating actually is and that is a big challenge in itself. I have another article about the flaws of decomposition based estimating and process optimisation (ie creating a work breakdown structure), but thats going into a book instead so you are gonna have to wait 🙂
regards
Paul
I reckon I would mostly stick with my estimate rather than take the marble, not because I have a pessimist bias but because I have proved my gut-feel estimates to be right often enough to trust them 🙂
Plus picking the wrong marble is a definite lose, whereas if I stick with my estimate it’s up to you to prove me wrong 😀
Interesting rationale there. This is obviously going to be opinionated, but George W Bush went with his gut too and look at the result of that. But your right in that respect. Has he been proven wrong? Depends on who you ask.
The cognitive bias about your gut-feel estimates being “right” is also subject to a proven cognitive biases. Take your pick from:
Choice-supportive bias, confirmation bias and negavity bias (in particular).
Furthermore your “gut feel” is easy to test. If you took a series of true/false questions and your gut told you you are 90% accurate, then if say there were 50 questions, then 45 should be right.
If only 30 are right then your gut feel is not as accurate as you think it is.
if you retook this test with different over and over again, and you never reach the 45 out of 50, then your “gut feeling” is itself shaped very likely by the three biases I listed.
Take a look here:
http://en.wikipedia.org/wiki/List_of_cognitive_biases