{{ searchResult.published_at | date:'d MMMM yyyy' }}

Loading ...
Loading ...

Enter a search term such as “mobile analytics” or browse our content using the filters above.

No_results

That’s not only a poor Scrabble score but we also couldn’t find any results matching “”.
Check your spelling or try broadening your search.

Logo_distressed

Sorry about this, there is a problem with our search at the moment.
Please try again later.

Big Data may be tough on our technology stacks, but the real challenges lie elsewhere.

Promises. Promises. Big Data sure makes a lot of them.

Increase the effectiveness of your sales (or political) campaigns by using behavioural data to divide customers into micro-segments.

Improve brand perception by monitoring the complex web of conversations across Twitter, Facebook and other channels and then engaging carefully with key influencers.  

Analyse internal processes to find opportunities to reduce costs and increase responsiveness.

Sounds great, but is it real? Are people actually doing such stuff, or is it all vendor hype?

I’m in two minds. I’ve seen signs that the promises are meaningful. Real people can analyse real data and get useful insights from it, driving real business value.

For example, I’ve seen a university find ways to identify students who were at risk of failing. They could then provide the added support needed to help them pass. (Important if you care about people, but also from a business perspective: they only get paid for the ones who pass).

I’ve seen a truck manufacturer improve operations across its dealer network using the insight gained from vehicle breakdown records. I’ve seen a telco learn a lot about the type of products its customers want.

But these were only glimpses. People had started to explore their data and were getting the odd, random benefit. They were still falling well short of the grand promises. They ran into brick walls at least as often as they made headway. And I’ve seen other organisations give up almost as soon as they began.

What’s going on?

Part of the problem is hype. The tech vendors are using Big Data as a hook to sell stuff. That’s the way the tech industry works. Create a lot of buzz around a vague concept (read Wikipedia’s definition of Big Data: almost anything could fit this definition) and hope enough people will latch on to it to drive Big Sales.

That fuzz doesn’t help the rest of us. Worse, it pushes us to focus on technology. But the real challenges of Big Data lie elsewhere.

In retrospect, I’ve been dealing with Big Data for a long time, for more than three decades. I started as a geophysicist, processing oil and mineral exploration data.  

We were certainly doing Big Data then: we needed dedicated supercomputers to handle it all. From there I moved into satellite image processing, and more special purpose hardware. Next I was building customer data warehouses, using yet another generation of specialist boxes.

Now it’s the web and social media. But the hardware has changed – now we're using a cloud of commodity kit.

Somewhere along this journey, we switched emphasis. Until about a decade ago, we worked as generalist teams on specialist hardware. (I say “generalist” because everyone had to do a bit of everything – if you wanted to analyse data, you had to know enough about hardware and software to make it do what you wanted).  

After that, we formed ever more specialist teams – the web team, the customer insight team, the IT team, and so on – and started to use general-purpose hardware.

I’m not sure that was a useful switch.

Yes, Big Data has always challenged our hardware and software stacks. (That’s the definition of Big Data.) The complexities of modern cloud architectures and software stacks such as Hadoop certainly need a lot of skills to pull everything together. That drives a degree of specialisation within the team.

But the real challenge of Big Data has always been in integrating multiple perspectives. You need to generate meaningful business hypotheses about what might be going on. You need to map those hypotheses onto the underlying algorithms and statistical concepts in a way that makes sense. (It’s easy to create bizarre conclusions if you don't respect the limitations of the data and the algorithms).  

You need to meld together data from a host of different sources. And yes, you need to understand the technology well enough to make it all happen.

Integrating such a diverse range of skills is tough. Pushing everyone into silos just makes this integration a lot harder. Yet that’s just how most companies have chosen to organise themselves.

Before these companies gain the full benefits of Big Data, they’re going to have to think about how they get people to work together. Many organisations talk about cross-functional teams, but they rarely support them well.  

They expect people to report into functional silos. When they do create “teams”, they refuse to co-locate them. Their facilities police pounce on people if they shift the furniture around. They align rewards to individual rather than team objectives. And so on.

Maybe Big Data, with its promise of extremely attractive benefits, will provide the impetus we need to start thinking more about how we create cross-functional teams?  

In the meantime, most of the talk – all those tools and technology – is just pulling our attention in the wrong direction.

(Image credit: Wikipedia)

Graham Oakes

Published 24 January, 2013 by Graham Oakes

Graham Oakes helps people untangle complex technology, processes, relationships and governance. He is contributor to Econsultancy.

43 more posts from this author

Comments (3)

Jahan Zahid

Jahan Zahid, CEO & Co-Founder at WebSherpa

Hi Graham,

I agree with you that the biggest challenges which are facing 'Big Data' are not with the technology itself but more the ability to draw meaningful conclusions from the data.

An IBM study in 2011 revealed that 90% of the world's data had been created in the last two years alone. This explosion of data is indeed placing a greater emphasis on being able to analyse subsets of the data to the needs of individual stakeholders. For example, Aristotle International have collected a vast amount of data and are using this to great effect for carefully targeted political campaigns in the US.

I believe there is still much more to come. The emergence of specialists who can model and form well grounded algorithmic analysis will also continue.

Jahan

over 3 years ago

Avatar-blank-50x50

Danny

Jahan is correct, it has only been a few years that the mass have realised the importance of data.

The 'data balloon' will continue to grow and soar as more businesses benefit from accurate data research.

over 3 years ago

Avatar-blank-50x50

Darren Abela

Graham makes some home truths here. Big Data is just a Big Hypothesis until organisations reengineer the way they function. Those who are making the first changes to address their silo legacy will be the first to break away from the pack and reap the rewards in the long run.

over 3 years ago

Comment
No-profile-pic
Save or Cancel
Daily_pulse_signup_wide

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Daily Pulse newsletter. Each weekday, you ll receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.