Software estimation considered harmful?

A couple of days ago I read Peter Seibel's Blog-Post Software estimation considered harmful?.

In a nutshell, he writes that he was reading the book "Software Estimation: Demystifying the Black Art". The complexity and pre-conditions needed to produce software estimations with some of the described techniques really made him think, but in the end he was mostly sure that developers should do estimations.

Then he met his friend Marc (who is running a small internet startup, Wesabe) telling him, that he makes no estimations at all. He just tries to excite his fellow programmers for specific features and they then just do it: "Marc's point of view, [as] I take it, is that the only reason to do these big projects is because you have to, and if you have to, it doesn't really matter how long it's going to take", Peter writes.

This brings Peter to the fundamental question: "Why estimate?" and from this to the question "Why set targets?" In the end, he sees software estimations more like a communication tool. Funny question, eh?

I started programming in school in 1983 and I did my first professional (for money) programming job in 1987. I studied computer science and systems analysis and I am running my own software company since 1999 with over 50 programmers now in january 2008. In my whole life, I never had a customer who was contracting me or my company without asking for an estimation in advance. And of course they expect that you will deliver as planned. Only if you are your own customer, you maybe can get away without doing estimations.

So the answer for me to the question "Why estimate?" is "Because your customers will expect this from a professional company". But the answer to "Why set targets?" is (maybe surprisingly) "Because it is much more fun to work"!

When you want to write great programs you should look for people who are smart and get things done. Smart is important, because it simplifies communication. You maybe don't have to explain fundamentals and you can exchange ideas much easier, because they are understood faster. More signal, less noise. But smart people are not always getting things done. Getting things done is about making stuff work. Finding the most simple and elegant idea/solution to a problem and extending it from that point on. For me "getting things done" also means "reaching measureable goals".

Why is programming more fun, when you set and reach measureable goals (targets)?!

It gives you freedom. You don't have to spend so much time to actually manage the project. This means less administration and more programming time, because when you set measureable goals and everybody in the team knows what to do, you don't have to talk about this so much anymore. You need less coordination, because you don't have to dispatch daily work-packages to your team-members. They can do it themselves.

You don't have to complete the work of others anymore. If today you know how to get things done, but others in your team don't, then maybe you have to complete their work to finish the project in the end. If you set goals and work with people who know how to reach these goals, everybody can concentrate on their own work. Or at least you find out early, if others can't meet their goals. This leads to the next point.

You are always in control of the situation and you can adapt to problems very fast. If the time estimation granularity for each goal is not too small and not too big (from my experience less than 5 mandays, 2-3 mandays in the average), then problems in the implementation will pop-up very fast and you can adapt your estimations based on the new experience. You can help people having problems reaching their goals. It's some kind of a natural risk-management. Instead of just reacting to problems, you take your destiny in your own hands.

You are more focused. And this makes you faster. When you set goals and you reach them, then you know what's behind you, what has already been accomplished. Then you can focus on the future and what still has to be done. This is actually really satisfying. Having a measure of what has already been accomplished and what to do next.

Your customers are happy. Customers are happy, when you deliver as promised. And when your customers are happy, then you will be happy, too. They will recommend your service and they will pay your invoices fast. This leads to a positive cash-flow and this will make you independent. And independence is one ingredient of personal freedom.

This is the normal way to accomplish something: If you want to reach a goal, you should decompose the goal into a hierarchy of sub-goals. You estimate the time/costs required to solve the sub-goals and then you (the team) tackle sub-goal by sub-goal. When all sub-goals are completed, the top-level goal must be completed, too. This is how you plan the birthday party of your kids, it is the way to build a house and to plan a holiday trip. Building software is different from building a house, but it is possible to do planning. Why do so many people think, that you can't plan a software project?

Maybe because they have a misconception about the meaning of estimation and planning or maybe it is because they had a bad experience in their life as a programmer when they were forced by their managers to make unrealistic estimations and stick to them?!

You should see it like this: To estimate a software project does not mean, that you have to foresee the future. It just means that you are making time-based assumptions and predictions. And "Project management" means, that you check every day if your assumptions and predictions are still correct and that you have to refine your plans if something is going wrong. If you do this every day, your predictions get better and better and you don't stumble over so many surprises anymore.

Of course, you should be excited of what you are doing. But I think, that this is a pre-condition for every expert in some area: You are really good at stuff that you really like. I wonder how Mark is exciting his developers at Wesabe about not so exciting features. Excitement is not exactly a prioritization method.

My fellow programmers and myself couldn't live without setting goals anymore. We are setting these goals for ourselves, because it is so much more fun to work. And as a by-product we get estimations our customers can rely on: If it works for us, it automatically works for them, too.

In the last 10 years we developed our own software engineering methodology which is based on the assumption, that software project planning and estimation is really a combinatorical optimization problem. Since 1999 we delivered all of our projects on time.

Since about 2001 we have an internal system to manage our projects, features, estimations and releaseplans, called Captain Feature. So we are not only having years and years of experience in doing this, but we also have a big database of completed software projects together with their planned estimations down to the feature level and the actual time needed to do the implementation.

So, we have the data to support my theory: It is possible to estimate and plan a software project. And it can be done in a very natural, programmer- and at the same time customer-compatible way.

I will talk about this in more detail in a later posting.