After a cycle of hype, reality sets in. Big Data isn’t going to save the world.

People are starting to state the obvious. Big Data in itself has little value. Unless you connect it to organisational goals, it’s just another tech solution looking for problems to solve.

And even with that connection, Big Data can be a dangerous tool. It’s easy to generate meaningless correlations if you don’t think systemically about your problems.  

It’s easy to generate spurious results if you don't pay constant attention to data quality. You can waste a lot of money if no-one acts on your findings. Nope, Big Data really won’t save the world.

But we knew this all along. I’ve seen very little written about Big Data that didn’t stress the need to manage data quality, to build integrated teams, to understand the limitations of the algorithms (that’s what “data science” is all about), to frame good questions.  

If you’re not prepared to do all this, avoid Big Data.

So where’s the backlash coming from?  Partly, it’s built into the tech industry’s DNA. We put technologies through a regular cycle of hype then backlash before we eventually settle upon reality. Big Data has now entered the backlash phase.

And partly, I suspect we’ve now had time to see some disappointing projects. There are many ways to fail with Big Data.  

For example:

  • Focus on Big. People get focused on processing as much data as possible, rather than on using the data which best fits their questions. Yes, Big Data gives you tools to extract signal from noisy data, but that doesn’t mean you can create (valid) signals from irrelevant data.
  • Hammer think. As the saying goes, “If all you have is a hammer, then everything looks like a nail”. Making the hammer bigger doesn’t make it more widely applicable. Many people seem to be trying to apply Big Data indiscriminately.
  • Tech-driven fantasies. Hadoop is a great tool. It makes sense for most IT organisations to get some understanding of it, even run a few prototypes. But if Big Data is a tech-driven project, it will fail.  

    You need a balanced set of business, data and technical skills. Few IT departments can assemble that mix.

  • Random data walks. This is the data scientists’ version of the above. Data is fascinating – given enough of it, you can find endless correlations, generate innumerable hypothesis, go down any number of blind alleys. You need to bound this exploration with some sort of business rationale.
  • Short-termism. It’s often easy to generate a few quick wins. Generating sustained results requires much more attention to organisational change – embedding analysis into day-to-day decisions, changing processes, building skills. This is all hard stuff. It’s easy to get bogged down after a few early successes.
  • Failure to act. When Big Data initiatives get isolated from regular business operations, they produce insight that no-one is committed to act on. No matter how good your insight is, without action it’s valueless.

And what can you do to avoid such failures?  

Here are some things I’d keep in mind when kicking off any sort of Big Data initiative:

  1. Data informs judgement; it doesn’t replace it. Your goal is to make good decisions. Data helps you do that. But data alone can’t make the decisions: you need to exercise judgement somewhere.
  2. Ignore the hype. The hype drags us into tools and technologies. It pulls us towards new algorithms and “interesting” data sets. You need to cut through this and focus only on the data, tools and techniques that apply to your specific problems.
  3. Focus on value, not size. You need the right data to help make your decisions. If that data is noisy, then Big Data techniques may help you find signal. But that’s a secondary issue – the starting place is solving high value problems, not using lots of data.
  4. Integrate. Big Data initiatives have to deliver capabilities that can be integrated into daily business operations. They need integrated business, data and technical teams. Standalone Big Data is a waste of time.
  5. Ongoing experimentation, not repeatable processes. You’ll often need to iterate to find the best answer to any specific question – generate a hypothesis, choose data and tools to test it, gather feedback, and refine. Trying to define the “perfect” process for a decision too early cuts you off from that feedback and refinement.
  6. Manage data quality. That’s a really boring, old world thing to say. But it’s no less true for that. Big Data gives you some tools to accommodate noise in your data, but you’ll always get better results if you start with decent data in the first place.

The bottom line is clear: without data, we tend to make poor decisions. Read Thinking, Fast and Slow by Daniel Kahneman. We’re all subject to a range of biases and shortcomings that naturally limit our ability to make good decisions. Paying careful attention to data can help us overcome these shortcomings.

Big Data isn’t a panacea. It simply adds another set of tools to our bag. Used inappropriately, those tools will dazzle us with inconsequential results, overwhelm us with trivial findings, drain resources from more useful work.

But, used well, Big Data can help us make corporate decisions based on reality rather than fantasy. In many organisations, that’d be a big change.

Graham Oakes

Published 19 August, 2013 by Graham Oakes

Graham Oakes helps people untangle complex technology, processes, relationships and governance. He is contributor to Econsultancy.

43 more posts from this author

You might be interested in

Comments (3)



about 5 years ago



It's still far too early to be so excited and a refreshing point of view that illustrates how crucial people still are.

almost 5 years ago


Guy Cuthbert, Managing Director at Atheon Analytics

As Oakes clarifies, people are starting to state the obvious; big data in itself has little value. There has been a focus on big data for some time now, however organisations have been missing opportunities that lie with the wealth of data that they already possess. In other words, it’s not about the big, it’s about the data.

There is an opportunity to transform the way in which businesses make sense of and gain value from data, if they hold the belief that rich, well-organised data enables strategic vision into business operations; and this involves three key points:-

Firstly, all data worth keeping should be treated as a business asset; protected, nurtured and curated such that it is available in a timely, reliable and consistent manner.

Secondly, business reporting should be focused on addressing genuinely *key* performance indicators, addressing Donald Rumsfeld’s ‘known knowns’:

“There are known knowns; there are things we know that we know, there are known unknowns; that is to say there are things that, we now know we don't know. But there are also unknown unknowns – there are things we do not know we don't know.”

Effective reporting highlights important metrics, and puts them in the context of what the organisation knows, understands or expects: previous performance, forecasts, budgets and plans. This allows directors and employees to contrast actual business performance with what is expected. Business reporting, however, does not address effectively the challenges of the ‘known unknowns’ or the ‘unknown unknowns’; to do so requires the effective application of data analytics.

So, thirdly, businesses need to apply analytics to their data – of any scale – in order to explore, discover and understand their ‘known unknowns’; those business questions that are well formulated, but whose answers are unclear. Effective analytics will also yield, from time to time, surprising ‘unknown unknowns’; genuinely new insights into business performance in areas which were not under active investigation.

As Oakes suggests, we need to focus on value, not size, and that starts with making the most of the data that we already have.

Visual analytics is the swiftest and most accessible route into data which utilises our visual perception system and its innate pattern discovery; no-one needs a degree in statistics or mathematics to be an effective visual analyst. The best of the current breed of visual analytics software encourages exploration of data, of all forms and sizes, and enables simple, effective communication of insights discovered to non-technical audiences. So yes, big data isn't going to save the world, but was it really ever going too?

almost 5 years ago

Save or Cancel

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Digital Pulse newsletter. You will receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.