{{ searchResult.published_at | date:'d MMMM yyyy' }}

Loading ...
Loading ...

Enter a search term such as “mobile analytics” or browse our content using the filters above.

No_results

That’s not only a poor Scrabble score but we also couldn’t find any results matching “”.
Check your spelling or try broadening your search.

Logo_distressed

Sorry about this, there is a problem with our search at the moment.
Please try again later.

Marketing now is as much about getting buy-in as it is about doing, well, 'marketing'.  

And in order to get buy-in, marketers need to do presentations - a lot of them, according to big brand marketers.

In these presentations, marketers need statistics about platforms, technologies, countries.  

And having interesting statistics helps marketers tell their audience that they are in touch with what's going on in the outside world.

Statistics have other uses, too

Statistics are also useful for spotting trends and getting new ideas. Econsultancy publishes statistics frequently and they are some of our most popular posts.  

And for subscribers, we pull these into an Internet Statistics Compendium, so that you can get a ton of them all in one place.

But...

But statistics can also cause problems for marketers. There is just so much information to wade through on the internet, that it can be hard to find the one piece of data which backs up an otherwise flawless story.

And when deadlines loom, it's quite easy to just cut and paste the best-looking chart into your presentation and hope for the best.

This approach, however, has led to a lack of rigour in marketing presentations and the spread of misinformation.  

And these bad statistics can easily come back to haunt you when things aren't working as well as you predicted.

So to help out, we've come up with three simple steps you can take to avoid bad statistics. And to illustrate why, there's a short case study about a widely reported marketing statistic that looked rather suspicious...

So, how can marketers avoid bad statistics?

1. Look for the primary source

Before dropping a chart into your presentation, always have a look at the source of the data.

That is, where did the data come from? Is it a reputable source? Or a survey by a company you have never heard of before?

And if you can, have a look for the original publication to see how the publisher obtained the data and whether it is still relevant to your purpose.

You may find that the data comes from a US-only survey, for example, or that it was gathered five years ago.

Facebook users example

Recently I needed to know the Facebook reach in Asia-Pacific, and found three sources which claimed that Facebook had between 3.6m and 3.8m users in Singapore.

Internet World Stats (3.6m)

Hashmeta (3.8m)

We Are Social (3.63m, when added up)

With a little investigation, I could see that they all used the same source: Facebook-reported data.

As Facebook makes its audience data available to advertisers, it was surprisingly easy to check.

Facebook (3.5 - 4m)

And, yes, the primary source verified what the others had reported - tick.

2. Use your own intuition

Once you know the primary source and are comfortable with the methodology, then it's prudent to review it against what you yourself know about the area.

That is, if you have some knowledge of a market or a platform, it can be worthwhile to see whether the data passes the sniff test before passing it on.  

Often if it seems to good to be true, you will find that the publishers used an obscure source or misrepresented the question in some way.

Back to Singapore and Facebook

Now I had no issue with Facebook as a data source. But, I had some idea of the entire population of Singapore (just over 5m) and it seemed far-fetched that 3.6m, or around 70%, of the population had a Facebook account.

And, as Facebook only reports for people over the age of 13, the amount would be even higher, more like 8 out of 10. This seemed very unlikely and required further investigation.

3. Find additional sources

If you are suspicious about a data point or you are using the statistic for something important, like a business plan, then you should try to find a second source.

And nowadays this is relatively easy. A short search on Google should turn up other reports which have a similar statistic. And if it's close, then job done.

But you may find that everyone is using the same figure, in which case you may have to dig a bit deeper in order to verify what you are going to include in your presentation.

Singapore sleuthing

So, searching again I found that most sources were using Facebook's published data. But I know that Singapore meticulously keeps and publishes stats about its population.

Instead of looking at the whole population though, I decided to look at one segment, 20 to 29 year olds, for which both Facebook and Singapore publish figures.

Facebook said that it had 1.2m people between the ages of 20 and 29, inclusive. But on the Singapore government site there are only 535,000 people in that age bracket.

Facebook, Singapore reach for age 20-29 from Power Editor (1.2m)

Singapore government statistics, people aged 20-29 (535,000)

What about non-residents?

Now, the remaining could be non-residents. But again, Singapore keeps close watch on these numbers as well.

Unfortunately Singapore does not break out foreigners into age brackets, but it does say that there are 1.63m non-residents in Singapore.

In order to make up the difference between the Facebook reported figure and the official statistics for people aged 20 to 29, though, there would need to be 700,000 foreigners between the age 20 and 29 in Singapore on Facebook. 

So for the Facebook population to be as reported:

  • 43% of all foreigners in Singapore must be between 20 and 29 (unlikely)
  • 100% of them must be on Facebook (unlikely)
  • 100% of all Singapore citizens between 20 and 29 must be on Facebook (unlikely)

This was starting to sound suspicious. And note that all it took was: 

  • A look at the source of the statistic
  • A quick thought about whether it was realistic
  • Research on a secondary source

A more realistic figure

Instead, if we used:

  • A generous estimate of 1/3 of expats as between 20 and 29 (538,000)
  • Added them to the Singaporeans aged 20 to 29 (530,000)
  • And then estimated, also generously, that 90% of the total (1,068,000) are on Facebook...

...then we would still come up with only 962,000 people between the age of 20 and 29 have accounts on Facebook, 22% less than reported.

What to make of this

Of course, it's easy to dismiss this as a small error which doesn't matter in the larger scheme of things.

I mean, figures are often inflated and people should know they are just estimates, right?

But as we all know, these numbers are then included in other estimates. So then this error is repeated and compounded and, in the end, we may very well be using some bad statistics as justification for our strategy.  

And if we do that, then we shouldn't be surprised if what we propose doesn't work out as well as we projected!

So why are the numbers inflated?

I'm not entirely sure. It could be a sample error, a 'creative' way of counting or even fraudulent users.

But what's important is that by all logical reasoning the reported and repeated statistic is not correct. And the more it is repeated, the more likely it is to be believed.

And if this number isn't correct, what does that say about the rest of the statistics? Seems to me that they probably all need revisiting in order to avoid knock-on bad analysis.

I mean, imagine if the next estimate was a 22% undercount of Facebook users. There would be headlines that Singapore youths are leaving Facebook in droves!

So...

So, do have a closer look at the statistics you are using and make sure that you  

  • Know the source
  • Think about whether it makes sense
  • And find a back-up source in case it doesn't look right

You will be then be passing on good, reliable figures and your predictions are more likely to be true.

And who knows, with just a bit of work you may even come up with a different solution which is against the trend, and as a result much more effective!

Jeff Rajeck

Published 20 January, 2016 by Jeff Rajeck

Jeff Rajeck is the APAC Research Analyst for Econsultancy . You can follow him on Twitter or connect via LinkedIn.  

110 more posts from this author

Comments (1)

Deborah Kay

Deborah Kay, CEO at Digital Discovery

Have none of the above four mentioned responded? Facebook? We are Social? Hashmeta? Internet World Stats?

I haven't studied the Facebook numbers in detail but your questions are valid and I'm not surprised, though I thought Facebook culled a lot of fake accounts in Mar 2015.

One possibility is under-13 children faking their age to open a Facebook account (unlikely). Or the older age groups using a false birth date (29 forever! more likely). Without age verification, that data should be taken with a pinch of salt.

The 20-29 range will definitely overstate usage - most say they are on Facebook but don't use it much. I have also seen complaints about how Facebook defines "active" users.

With foreign workers 100% penetration is possible, if they have multiple Facebook accounts or if there are clickbait farms creating fake accounts from Singapore. I've played around with audience insights before where if you filtered the data in a certain way, the top page for Singapore becomes the Philippines Embassy. Most Philippines I know have dual sim-cards even in the Philippines so who knows if they have multiple Facebook accounts.

My next question is whether the data is more skewed in Singapore vs ROW. An Adweek report says Singapore leads the way globally with 86% of suspicious mobile traffic.

If you think about it, why would Singapore have the the highest CPC and CPM (3X higher than APAC average) - it's so small so can't be 3X more competitive than Australia. Another eConsultancy report shows that marketers are not satisfied with metrics.

Are we in a prisoners dilemma where brands don't see results and agencies+FB convince them that they need to spend more to see results? Everyone benefits except for the client.

Or are we in a vicious cycle where fake likes drive CPC up, causing people to buy more fake clicks, and once again driving CPC up?

And why would Facebook care? They're laughing all the way to the bank....

(FYI Not a marketer, so these are just observations and hypotheses. I could be wrong on all counts)

5 months ago

Comment
No-profile-pic
Save or Cancel
Daily_pulse_signup_wide

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Daily Pulse newsletter. Each weekday, you ll receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.