So why do organisations keep doing projects?

It could be insanity. They keep doing the same thing over and over again, hoping for different results. Some organisations certainly look like pretty close to the edge.

But I don’t think that’s the case. I think most organisations are getting a pretty fair return from their projects. Individual projects may be painful but, across the entire portfolio, the benefits outweigh the costs.

All the discussion of failure is blinding us to something else. You can have many failures, yet still deliver a lot of value.

I learned this in my first job. I started out as a geophysicist, working for an oil company. Oil companies have lots of failures.

When I graduated, the statistics said that eight out of nine oil wells failed, i.e. they didn’t discover commercial quantities of oil. That statistic had held for decades and, as far as I can work out, it still holds today.  As we get better at exploring for oil, we start seeking it in ever tougher environments.

So for decades the fundamental oil industry project has had an almost 90% failure rate. Does that mean that the oil industry has failed?

Not so far as I can see. It’s built some of the world’s largest companies. It’s created a lot of wealth. You can question some of its actions, and wonder about its future in a time of global climate change, but by the standards society tends to judge these things, the oil industry is a pretty clear success.  

The small number of successes has outweighed the large number of failures.

The fact is, simple statistics about the percentage of projects that fail tell you little about the likelihood of organisational success. You need to drill deeper if you want to do more than feed the project failure industry.

For a start, there are different types of failure. I see two distinct types, which I call inherent and unnecessary.

Inherent failure

Inherent failure is about risk.  When you set up a project, you take on risks. In particular, you take on domain-specific risks. In the oil industry, these are risks arising from geological uncertainty, inherently ambiguous geophysical data, etc. In the games industry, there are risks about game mechanics and gamer engagement. And so on.

You can’t avoid these risks. You can only mitigate them within certain limits. The risks are inherent to the industry you’re working in. They’re built in.

You can’t avoid inherent failure. You have to manage it.  You bring domain expertise to bear to reduce it where you can. You apply statistical, portfolio management techniques to survive and thrive amongst the residual risk.

Inherent failure doesn’t have a lot to do with project management. Project managers ensure that the right domain expertise is brought to bear, that this expertise isn’t hampered by poor logistics or suchlike problems, that it has the tools it needs.  

They provide portfolio managers with the information they need to manage portfolio risk. Then they get out of the way.

Unnecessary failure

Unnecessary failure is what happens when you don’t get the basics right. Most of the problems that get trotted out over and over again – poor communication, weak sponsorship, bad estimating, inept scheduling – fit into this category.

It is possible to get these things right, or at least right enough. People do it more often than we (or the project failure industry) give them credit for.  But getting them right doesn’t make projects succeed – it simply gives you space to concentrate on the inherent risks.

The project failure industry only concerns itself with unnecessary failure. Its prescriptions rarely say anything about how to deal with inherent failure  But it lumps inherent and unnecessary failure into the same statistics to help create a market for its snake oil.

Managing inherent failure is tough. You need specialist expertise, attuned to the specific industry you’re operating in. You need to take risks  (If you don’t drill any wells, you’re guaranteed to not find any oil).

And because you’re taking those risks, you need to apply portfolio risk management techniques. That means monitoring projects carefully, shifting resources to where they’ll add most value, killing projects early if they can’t succeed. (Most organisations are incredibly bad at killing projects).

The project management industry talks a lot about unnecessary failures. That’s fair enough: we need to eliminate them.  

But as we eliminate them, a large pool of inherent failures is going to remain. We need to get better at managing them too.