Does anyone else use Alexa.com for (crude) comparative traffic analysis? With a little joint effort could we figure out just how accuracte / inaccurate its traffic data is?

If you haven’t seen or used Alexa before, have a look at E-consultancy’s site traffic details over the last year. Clearly as the graph looks positive I’m happy to give this link ;). You can also compare your site’s traffic details with that of other (competitor) sites. 

On the face of it this would seem to be a free service to help you benchmark your own traffic and, more interestingly, that of other sites e.g. competitors, potential partners etc. A free version of Hitwise - though of course Hitwise does a lot more.

Certainly I have heard lots of people, even owners of large sites with plenty of budget for more sophisticated media planning tools,  say they use Alexa even if just for an intial indicative probe into another site.

But how accurate are these rankings? Alexa has a page which explains how its rankings are calculated.  This includes the following disclaimers:

• Our users are disproportionately likely to visit alexa.com, amazon.com and archive.org, and traffic to these sites may be substantially overcounted.
• The Alexa Toolbar works only with the Internet Explorer browser. Sites frequented mainly by users of other browsers will be undercounted. For example, the AOL/Netscape browser is not supported, which means that Alexa collects little data from AOL users, and our traffic to aol.com is likely lower than it would be for a more representative sample.
• The Alexa Toolbar works only on Windows operating systems. Although a large majority of the Internet population currently use Windows, traffic to any sites which are disproportionately visited by users of other operating systems will be undercounted.
• The rate of adoption of Alexa software in different parts of the world may vary widely due to advertising locality, language, and other geographic and cultural factors. For example, to some extent the prominence of Korean sites among our top-ranked sites reflects known high rates of general Internet usage in South Korea, but there may also be a dispropotionate number of Korean Alexa users.
• In some cases traffic data may also be adversely affected by our "site" definitions. With tens of millions of hosts on the Internet, our automated procedures for determining which hosts are serving the "same" content may be incorrect and/or out-of-date. Similarly, the determinations of domains and home pages may not always be accurate. When these determinations change (as they do periodically), there may be sudden artificial changes in the Alexa traffic rankings for some sites as a consequence.
• The Alexa Toolbar turns itself off on secure pages (https:). Sites with more secure page views will be under-represented in the Alexa traffic data.

So my take is that for a UK site the rankings will be overly US skewed and that the rankings will also favour sites whose users include early-adopter / internet-y types. Sites whose target users are ’mass market’, and who are less likely to install the Alexa toolbar, will fare less well.

All of which means that E-consultancy is likely to be ranked much higher than it is due.

However, can you not compare like sites with like with some degree of accuracy? So sites which have the same, or very similar, target market? The sample biases might at least then be representative?

I’ve talked to people who’ve used Alexa in conjunction with other tools and, on the whole, they’ve been surprised with the apparent degree of accuracy - at least relatively speaking - that they get with Alexa.

Out of interest, I compared totaljobs.com with workthing.com on Alexa with totaljobs.com coming out a fair bit higher. Then I had a look at published audit data for the same periods on ABCe:

• ABCe audit for totaljobs.com 01/10/03 - 31/10/03
• ABCe audit for workthing.com 01/10/03 - 31/10/03

Looks to me like a fairly accurate correlation?

According to our web traffic analysis we average around 3,500 unqiue users a day generating around 10,000 page views and our Alexa ranking is around 17,000. Anyone else happy to share their stats (particularly sites likely to have a similar user skew) to see what the correlation is?

Does anyone else have views on Alexa’s accuracy or lack of? Do people use Alexa as a (crude) benchmarking tool?

Ashley

Last time I used Alexa was on or around 2001. I have compared their information with our actual traffic and it was clear that they were way off both in terms of absolute and relative figures (as compared to available information about our main competitor at the time).

Can Alexa be accurate in 2004? I am not sure thats the case because they are certainly not the most common toolbar out there, and more importantly I think people who would go for that toolbar would tend to relate to group whose browsing pattern may differ significantly from other groups (ie I will never install Alexa at home).

Would be interesting to compare relative figures from Alexa and HitWise, I'd do it but our subscription expired :(

Mmm... There are many factors for (large amounts of) error. We're assuming, of course, that the site stats figures given are correct which we all know isn't always the case - actually, quite the opposite in many cases still.

As a starting point I think you'd have to compare only sites with a) high enough levels of traffic (Alexa < 100,000) and b) pretty similar user bases (because of the toolbar skew) and then you'd need to be pretty confident that the site's unique user / page view stats were accurate.

Ashley

On 17:35:34 14 June 2004 textor wrote:

Not very.  A long time I did a graph

http://www.textor.com/alex.gif

comparing known traffic levels with alexa ranking.  Left side is visitor sessions per month, x axis is ranking. 

At the interesting end the graph is so steep that what looks like a close fit is in fact miles out.  I would expect a closer fit at the left hand end because the sample is bigger, but it is much harder to get any decent stats. 

If anyone on the list is interested in sending me stats / alexa rankings (no need to say what site it is)  I am happy to correlate and graph.

- only alexa < 500,000 need apply
- send visitor sessions per month and alexa ranking
- subject Alexa ranking data

Bob Browning
Textor Webmasters Ltd

Ashley,

As per our offline discussions, Alexa can be useful if used to analyse trends; but also dangerous if used as an absolute means of comparing sites.

At www.Top-Consultant.com we use Webtrends to track our site stats and I’m quite happy to share our average weekday traffic which ranges from 6,500 to 10,000 unique visitors (higher on newsletter days). So in principle our ranking should be quite a lot higher than e-consultancy’s; but in reality we have a lower ranking (which should have alarm bells ringing!).

The fact that e-consultancy is read by people at the cutting edge of e-business would imply that you are more likely to have readers who use tools like Alexa; and you’re also more likely to have a strong US readership. These are the types of factors that can explain such apparently perverse rankings.

The implications of this are straightforward... as soon as there are reasons to think that the demographics of two sites differ significantly, then a tool like Alexa cannot be used to square up one site against the other. So Workthing vs Totaljobs works quite well, but only because the reader demographics are VERY similar. E-Consultancy’s ranking is not that far off Workthing’s (last time I checked), yet your stats would confirm there is a world of difference in the actual visitor levels the two sites receive. Again this is down to the difference in your readership demographics

* So when is Alexa useful? *

Alexa can be useful for monitoring site traffic trends for one particular site - and for generating a list of similar sites by seeing which other sites Alexa users read once they’ve read the site you are checking up on. The average page views data is also much less likely to be skewed by the Alexa readership bias, so can be used to compare different sites. Above all, though, I would say Alexa is useful for checking on any site you are thinking of partnering with / advertising on. If a site does not appear in the top 100,000 sites then for us that sets alarm bells ringing and we would not proceed any further with that site without seeing independently produced / audited site stats.

Does this tie-in with other readers’ experiences?

On 11:34:12 14 June 2004 Ashley wrote:

Does anyone else use Alexa.com for (crude) comparative traffic analysis? With a little joint effort could we figure out just how accuracte / inaccurate its traffic data is?

If you haven’t seen or used Alexa before, have a look at E-consultancy’s site traffic details over the last year. Clearly as the graph looks positive I’m happy to give this link ;). You can also compare your site’s traffic details with that of other (competitor) sites. 

On the face of it this would seem to be a free service to help you benchmark your own traffic and, more interestingly, that of other sites e.g. competitors, potential partners etc. A free version of Hitwise - though of course Hitwise does a lot more.

Certainly I have heard lots of people, even owners of large sites with plenty of budget for more sophisticated media planning tools,  say they use Alexa even if just for an intial indicative probe into another site.

But how accurate are these rankings? Alexa has a page which explains how its rankings are calculated.  This includes the following disclaimers:

• Our users are disproportionately likely to visit alexa.com, amazon.com and archive.org, and traffic to these sites may be substantially overcounted.
• The Alexa Toolbar works only with the Internet Explorer browser. Sites frequented mainly by users of other browsers will be undercounted. For example, the AOL/Netscape browser is not supported, which means that Alexa collects little data from AOL users, and our traffic to aol.com is likely lower than it would be for a more representative sample.
• The Alexa Toolbar works only on Windows operating systems. Although a large majority of the Internet population currently use Windows, traffic to any sites which are disproportionately visited by users of other operating systems will be undercounted.
• The rate of adoption of Alexa software in different parts of the world may vary widely due to advertising locality, language, and other geographic and cultural factors. For example, to some extent the prominence of Korean sites among our top-ranked sites reflects known high rates of general Internet usage in South Korea, but there may also be a dispropotionate number of Korean Alexa users.
• In some cases traffic data may also be adversely affected by our "site" definitions. With tens of millions of hosts on the Internet, our automated procedures for determining which hosts are serving the "same" content may be incorrect and/or out-of-date. Similarly, the determinations of domains and home pages may not always be accurate. When these determinations change (as they do periodically), there may be sudden artificial changes in the Alexa traffic rankings for some sites as a consequence.
• The Alexa Toolbar turns itself off on secure pages (https:). Sites with more secure page views will be under-represented in the Alexa traffic data.

So my take is that for a UK site the rankings will be overly US skewed and that the rankings will also favour sites whose users include early-adopter / internet-y types. Sites whose target users are ’mass market’, and who are less likely to install the Alexa toolbar, will fare less well.

All of which means that E-consultancy is likely to be ranked much higher than it is due.

However, can you not compare like sites with like with some degree of accuracy? So sites which have the same, or very similar, target market? The sample biases might at least then be representative?

I’ve talked to people who’ve used Alexa in conjunction with other tools and, on the whole, they’ve been surprised with the apparent degree of accuracy - at least relatively speaking - that they get with Alexa.

Out of interest, I compared totaljobs.com with workthing.com on Alexa with totaljobs.com coming out a fair bit higher. Then I had a look at published audit data for the same periods on ABCe:

• ABCe audit for totaljobs.com 01/10/03 - 31/10/03
• ABCe audit for workthing.com 01/10/03 - 31/10/03

Looks to me like a fairly accurate correlation?

According to our web traffic analysis we average around 3,500 unqiue users a day generating around 10,000 page views and our Alexa ranking is around 17,000. Anyone else happy to share their stats (particularly sites likely to have a similar user skew) to see what the correlation is?

Does anyone else have views on Alexa’s accuracy or lack of? Do people use Alexa as a (crude) benchmarking tool?

Ashley

Thanks for replies - the message I'm getting is, as expected, that Alexa is useful in certain situations as a broad indicator but can't be taken too much at face value.

What other tools / services do people use in a similar way? Some obvious candidates might be Google PageRank? Or number / quality of links to a site? Or simply search rankings on relevant phrases?

Ashley

On 11:02:11 15 June 2004 textor wrote:

I think you are spot on in identifying Alexa as being a broad indicator that can warn you that something is afoot.   If someone asks me how far it is to the pub, the fact that my answer could be only a rough guess still makes it better than nothing.  Actually a lot better than nothing. 

Take our b2b site www.bobsguide.com whose alexa ranking has varied between 50,000 and 100,000.  That alone says the figures are dodgy.

But along comes Mr Client who says they would rather place their ads with http://www.btnsn.com/ .  It is very useful to me to know that their alexa ranking  is around a million. 

In the world of high traffic b2c sites you can rely on the figures in the media packs being pretty much OK.  Once you get to world of niche b2b sites things are different.  There are many sites from big important publishers who’s alexa ranking is in seven figures. 

http://www.bankingtech.com/
http://www.risk.net/

Ads on these specialised sites are not sold on a CPM basis.  I remember having an interesting conversation with one site (who shall be nameless) where I had to explain the difference between hits and page impressions to the salesperson.    They do not publish figures because they are (presumably) nothing to write home  about. 

That doesn’t mean that you should not advertise with them.  If you are selling software at £2million a pop then you are in a niche that can be very small.  Risk magazine is very prestigous and  the web site is very valuable real estate.  Alexa rating or not.

There are of course sites in our niche that have respectable figures.  Alexa is a way of sorting out the serious competition from the ones we can cast doubt on.  And because Alexa is free we can let the prospect see for themselves.

Bob Browning
Textor