First off, what is it?

Well don’t let anyone tell you it’s down to sample size, or about measuring everything. It’s about combining datasets (sometimes ‘dirty’ ones), contrasting them in different ways, and doing it as quick as possible.

Sometimes this necessitates great computing power, but not always. You can read more about such technology as Hadoop and GreenPlum in this nice little article).

Datasets are multiplying as we measure lots more than we used to. This means our thinking has to broaden – no longer is ‘what can we do with our database of email addresses?’ the question, rather ‘what data can we look at to give us the best idea possible of a customer’s stage in the buying cycle and what they’ll be receptive to next?’

The definition of big data isn’t really important and one can get hung up on it. Much better to look at ‘new’ uses of data.

So, here’s some examples of new and possibly ‘big’ data use both online and off-.


This article from the Wall Street Journal details Netflix’s well known Hadoop data processing platform. 

Cloud architecture is highly scalable and allows Neflix to quickly provision computing resources as its sees the need. Traffic patterns are analysed across device types and localities to help improve the reliability of video streaming and plan for growth.

The technology is also used for Netflix’s recommendation engine based on a customer’s viewing habits and stated preferences.


Sticking with Netflix, this piece in the Washington Post theorises that Netflix could vary its price if it had enough information on each user to know how much they might pay. 

To a certain degree, this happens in online retail with airlines targeting previous browsers, and some stores (such as Staples) changing prices depending on which physical store the customer is nearest.

The Wall Street journal has also documented that Orbitz, the travel website, has in the past charged Mac visitors higher than those on Windows. Taking into account IP, device, age, past visits, and more variables, throwing them into a database and calculating a charging threshold can conceivably be termed big data.

Out of home advertising

I’ve previously covered Route, who have combined lots of data on footfall and traffic, including the tracked day-to-day movements of 28,000 people. It’s hoped that the accuracy of predicting eyes on billboards will increase, leading to fairer pricing.


Retail Habits

I can’t recommend too often that you read this piece from the NY Times on how Target uses a wealth of customer data to predict future purchasing habits. Specifically, pregnancy kicks off a chain of purchases that are fairly distinctive – Target’s data collection is spookily prescient, sending one teen customer nappy vouchers before her own father knew she was pregnant.


Politics has traditionally seen data siloed, with canvassing done on little more than a list of postcodes. Obama’s election campaigns began to change this. Check out this article from Slate on Project Narwhal.

Narwhal would bring new efficiency across the campaign’s operations. No longer will canvassers be dispatched to knock on the doors of people who have already volunteered to support Obama. And if a donor has given the maximum $2,500 in permitted contributions, emails will stop hitting him up for money and start asking him to volunteer instead.

Those familiar with Narwhal’s development say the completion of such a technical infrastructure would also be a gift to future Democratic candidates who have struggled to organize political data that has been often arbitrarily siloed depending on which software vendor had primacy at a given moment.


WeatherSignal works by repurposing the sensors in Android devices to map atmospheric readings. Handsets such as the Samsun S4, contain a barometer, hygrometer (humidity), ambient thermometer and lightmeter. 

Obviously, the prospect of millions of personal weather stations feeding into one machine that will average out readings is exciting, and one that has the potential to improve forecasting.

You can read about OpenSignals work in this article in Scientific American.


Heart Disease

This piece from Cloud Times details how IBM are predicting heart disease with big data. Analysis of electronic health record data could reveal symptoms at earlier stages than previously.

IBM uses the Apache Unstructured Information Management Architecture (UIMA) to extract the known signs and symptoms of heart failure from available text. 

With no single strong indicator, only weak signals or ‘co-morbidities’, such as hypertension, diabetes, associated medications, ECG and genomic data etc. can be analysed. Drawing out probabilities from disparate and size-differing databases is a task for big data analytics. 

Infectious diseases

Again IBM, this Venture Beat article looks at a model and data from the World Health Organization. IBM looked at local climate and temperature to find correlations with how malaria spreads. This analysis is used to predict the location of future outbreaks. The Spatio Temporal Epidemiological Modeler (STEM) is free and open source.

Justin Lessler of Johns Hopkins Bloomberg School of Public Health:

There are a lot of tacit assumptions out there about how changes in climate will impact the distribution of diseases like malaria. This work suggests that things probably are not so simple. A change that has a huge effect on malaria transmission in one place might not be as important somewhere else.

Doctor performance

Crimson is a system that shows variables including complications, hospital readmissions and measures of cost. It colour codes signals as to how well a doctor is performing against his or her peers.

This piece in the Wall Street Journal suggests the technology has reduced average stay and average cost at the Long Beach Memorial Hospital. 

One doctor was warned by a pharmacist that data showed one physician was using Levaquin, an antibiotic, at a far higher rate than peers. With concerns about generating drug-resistant bacteria, the physician was encouraged to reduce the usage of said antibiotic. 

This particular medical group has used big data to make sure medication is correctly prescribed. 2012 data showing 76% of patients getting recommended shots compared with 56% in 2010.

10th use?

I’ll let you help me out with that in the comments.

Ben Davis

Published 16 October, 2013 by Ben Davis @ Econsultancy

Ben Davis is Editor at Econsultancy. He lives in Manchester, England. You can contact him at, follow at @herrhuld or connect via LinkedIn.

1246 more posts from this author

You might be interested in

Comments (15)

Save or Cancel
Doug Kessler

Doug Kessler, Director at VelocitySmall Business Multi-user

Great post.

For a tenth, didn't Google predict a flu epidemic by looking at search patterns for 'flu'? Not sure that's officially Big Data but it's big and it's data.

And my mother varies what to serve with the turkey at Thanksgiving based on the noises people made at the last Thanksgiving multiplied by the amount left in the dish divided by the amount in the dish at the start of the meal times Pinterest mentions minus preparation effort as a percentage of available preparation help -- all converged with commodity pricing in the mid-west, weather patterns and the degree to which she can be arsed. I could show you the algorithm but you'd have to come to Thanksgiving and I don't think you want to do that.

Also Data Sift has done some cool things predicting box office success before a movie's opening weekend.

almost 5 years ago

Ben Davis

Ben Davis, Editor at EconsultancyStaff


If IBM could capture Mrs Kessler in a parodic 'smarter turkey' advert, I'm sure it wouldn't hurt the brand.

almost 5 years ago

Ben Foster

Ben Foster, Operations Director at is using big data to predict health trends. We have pulled our 17million + page impressions on health into a map which shows trending conditions over time:

We are planning to extend this so that the system can auto-tweet major trends and changes, and also show a slider timeline of an "outbreak" over a 1 month period.

almost 5 years ago


Brendan Cooper

Mobile advertising utilises a huge amount of data. The latest programmatic techniques bring together data from publishers, advertisers, ad tech and third-party data bought in, and draw inferences from that data using algorithms to match the right ads to the right slots.

This is done through an auction process that happens while the mobile page or app is rendering - literally, milliseconds for many advertisers to bid for that one slot and the successful bidder gets to place the ad. It's entirely possible that this auction happened to place the ads you're seeing on this page right now.

almost 5 years ago



Having a mainly retail background, I'm more than familiar with a few of these applications. But what really stands out is not the commercial application, but the medical application, where analysis of data really could be genuinely life-saving. In this age where everyone is so cautious about their personal data, I for one would be perfectly happy for my details to be 'out there' if it meant saving me from potential heart disease. I wonder how much more progress could be made in this field if it wasn't for people abstaining from allowing use of their personal details.

almost 5 years ago

Peter McCormack

Peter McCormack, Founder at McCormack Morrison

Isn't it time we started looking beyond Big Data?

I mean has anyone sat down and considered Really Big Data? It's the new Big Data!

almost 5 years ago

Ben Davis

Ben Davis, Editor at EconsultancyStaff


Love those heat maps - compelling and useful.

almost 5 years ago

Malcolm Duckett

Malcolm Duckett, CEO at Magiq

Seems to me that the maps needed to present "occurrence per thousand searches", as otherwise they look a lot like population density maps? (there are a lot of people in London, Manchester, Aberdeen etc...)

almost 5 years ago

Ben Davis

Ben Davis, Editor at EconsultancyStaff


That's a good point.

almost 5 years ago


Richard Beaumont

Then there is using big data, to look at how and by whom a lot of this big data is being hovered up:

almost 5 years ago

Ben Foster

Ben Foster, Operations Director at

@Ben @Malcolm thanks for the feedback always nice to get other people's input! We will consider this for future versions.

almost 5 years ago

dan barker

dan barker, E-Business Consultant at Dan Barker

1. This page uses Doubleclick, which gathers information about what I'm doing around the web, uses that to figure out my age & gender, and then - depending on how much money various advertisers offer them, and the particular data it's captured about me - displays ads in real time to me as I browse around other sites.

2. The page also uses Google+ which, if I move my mouse over that '+1' button, recommends content to me based on articles others in my 'circles' like on this site.

3. And of course, it uses Google Analytics, which tells you what I've done on the site, and can be aggregated up in any number of ways including at a member-type level based on info Econsultancy dynamically passes to it as the page loads.

All three of these are so commonplace that they feel really mundane, but I suppose you could categorise all 3 of those as uses of big data!

almost 5 years ago


Joe Crandall

One of the largest users of analytics is the energy industry. They analyze information from many disparate sources in order to predict energy consumption levels and the associated energy production requirements.

almost 5 years ago


Vicky Smith, Consultant - Online, Responsible & Volunteer Tourism at Freelance

Hey Ben,

Good article, and well done for including non-commercial uses in the medical and political field.

There's also a lot going on in disaster relief, bringing together crowdsouce mapping and satellite technologies to respond more quickly, efficiently, cohesively and better respond to disasters coordinating many international organisation and volunteer contributions. E.g.


over 4 years ago

Ben Davis

Ben Davis, Editor at EconsultancyStaff


Thanks for the comment and the link!

Disaster response is definitely an area that needs further investment and if big data can be utilised, all the better.

over 4 years ago

Save or Cancel

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Digital Pulse newsletter. You will receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.