CEO at Econsultancy
14 January 2003 14:44pm
Just a quick recommendation to any of you particularly interested in web measurement, e-metrics, analysis and reporting...
There aren't that many quality resources dedicated to this topic but I would recommend Hurol Inan's 'Attuned' newsletter which you can subscribe to at http://www.hurolinan.com/book/n_03.asp
Hurol is author of "Measuring the Success of your Website" and consults / speaks widely on the topic. The newsletter features vendors, with profiles and analysis, as well as recommended articles and books.
(also on the site is a good overview of web measurement vendors at http://www.hurolinan.com/book/br_list.asp?ResourceType='Vendor';)
Engagement Manager at Omniture
14 January 2003 16:51pm
I'd second that Ashley - and not just because my e-mail response is the featured article in this month's Attuned!
You are correct that quality resources dedicated to site measurement are in short supply. Readers would do well to visit emetrics.org, which is the result of last year's inaugural E-metrics Summit in Santa Barbara.
Attendees expressed a collective desire to "justify Web expenditures and quantify the results", which, as we all know, is a walk in the park ;-)
14 January 2003 16:58pm
My book was featured (and glowingly reviewed) in the latest 'Attuned' so I too have some vested interest to declare although I don't think you can access the archives so any new sign-ups won't benefit me...
We're working on a white paper/report on web metrics and will be featuring selected vendors. It will be out around March and available to our 8,600+ registered users. If you don't mind I'll contact you separately about this as Clickstream is on our list of vendors to feature.
15 January 2003 10:51am
Get in touch whenever it is convenient.
I followed up my e-mail to Hurol with another, which explains the frustration of client-side tracking vendors such as Clickstream.
We pick up a client, previously using a server log analysis product and start producing Page Impression figures anything from 20% to 80% higher than those reported by the log analysis tool. The usurped vendors are ABCe associates - as are we - and the client cannot understand how the previous figures and ours can both be 'right'. We explain the principle of auditing and the fact that it is the tool and the methodology rather than the figures themselves which are audited, but this rarely cuts any ice with the client.
The result is the client having to explain to their advertisers why they are about to increase their placement charges and, when they next audit, having to explain to their market the sudden leap in online traffic.
There is potential here for the client to lose faith in the auditing process in particular and online measurement in general and, indeed, we have seen this happen on more than one occasion.
The client might be happy with us, but believe the whole web measurement area lacks credibility. So long as we in the analytics business are telling companies that they had 5 million page impressions last month...but that they also had 5.1,5.2....6, 7, 8 million, depending on the tool selected, I believe this situation can only deteriorate.
As a former employee of a log analysis tool vendor who has moved to what I consider best-of-breed, client-side tracking, I accept that I am somewhat less than objective when it comes the merits/demerits of each methodology :-). Leaving aside the question of which is superior, everybody in this space should at least be able to agree that we work towards some form of consistency for measuring headline metrics such as page impressions.
Making analytics work for any online business is not and never will be an exact science. However, with one or two caveats, the calculation of core metrics like page impressions can be. The better job everyone in the industry can do to convince end-users that their figures are genuine and credible, the easier it should be to encourage the same people to use these numbers as the foundation for extended metrics, ROI calculation and the like…pushing the envelope of web analytics in the way we all want to see this industry develop.
The logical first step in this process is to ensure we vendors are all singing from the same hymn sheet when it comes to first principles of how to measure simple metrics. How long can the industry sustain the notion that widely divergent figures produced by competing tools are both 'right'? Moreover, will organisations continue to accept such deviations in measurement indefinitely?
On 16:58:58 14 January 2003 Ashley wrote:
>My book was featured (and glowingly reviewed) in the
>latest 'Attuned' so I too have some vested interest to
>declare although I don't think you can access the archives
>so any new sign-ups won't benefit me...
>We're working on a white paper/report on web metrics and
>will be featuring selected vendors. It will be out around
>March and available to our 8,600+ registered users. If you
>don't mind I'll contact you separately about this as
>Clickstream is on our list of vendors to feature.
15 January 2003 11:06am
I agree (and applaud) what you say. It certainly has to happen, and market forces will inevitably ensure it does. Indeed, this topic was the cover story on NMA this week.
In theory I believe client-side measurement must be more accurate as there is so much less 'noise' created by the redundant data included in log files. However, I'm interested that you have found your client-side measurement to *increase* page impressions by so much - why do you think this is? Is it 'back' click page impressions that the log files are missing (and caching more generally)? Are the log tools being set up to screen too many pages out, or are they not logging everything?
What do you think are the main causes of discrepancy between the 2 techniques? On e-consultancy we're currently running 2 measurement tools (Webtraffiq and LiveStats) - one client-side and one server-side - and the results are indeed often quite different. (separately we do our own tracking and analysis which we have much more faith in to be honest)
16 January 2003 12:04pm
It's a combination of factors:
1 - Web server log files will fail to record to any content served by caching servers or proxies (one or two of the better tools can accommodate some proxy logs).
2 - They also fail to spot pages served from the local browser cache. This is increasingly THE problem as browser timeouts increase with every new version of IE.
3 - By definition, a log file writes a sequential record for each request made - meaning there are no two-point page timings.
4 - Although many of the better log file analysis tools will filter out 'noise' at point of database ingestion, the fact that every request for every resource is initially recorded, creates its own problems in terms of size, file transfer, sewing, and the general unwieldiness of it all.
I used to work for a well-known log analysis vendor and I know that they now include a tagging option to enhance the log file data. Great, but this simply demonstrates the inherent inaccuracy of log file tools.
As I said, even on dynamic sites, we have witnessed 20% caching of pages. On static sites, this figure has been considerably higher. Further, as a general rule of thumb, the more popular the page, the more likely it is to be cached (either by the site owner to enhance the visitor's experience, or by the browser as the page has been previously viewed). So it's not simply a question of the quantity of pages missed by log file tools, but the quality. An unusually low 5% caching might not be considered material until you discover that missing this 5% completely distorts your impression of the most popular pages on your site, perhaps rendering some of your most popular content some of the least.
Again, the most popular pages are often those that enjoy the longest page view durations (not always, but sometimes). So 5% caching of pages might translate to 25% of total time on site. We have bona fide experience of a client whose page caching was 40%, but these pages accounted for 80% of the total time visitors spent on site. A log file tool, whilst missing 40% of page views (which is bad enough), would have misappropriated 80% of page view duration to other pages.
Thus, the inability of log file tools to record cached content is a double-edged sword. Not only is there a shortfall in data, but the data recorded can be skewed out of all resemblance to reality.
If users of log file tools think they can simply up all their metrics by 20% (or whatever their techies are telling them is the incidence of caching on their site), then they are kidding themselves.
On 11:06:49 15 January 2003 Ashley wrote:
>I agree (and applaud) what you say. It certainly has to
>happen, and market forces will inevitably ensure it does.
>Indeed, this topic was the cover story on NMA this week.
>In theory I believe client-side measurement must be more
>accurate as there is so much less 'noise' created by the
>redundant data included in log files. However, I'm
>interested that you have found your client-side
>measurement to *increase* page impressions by so much -
>why do you think this is? Is it 'back' click page
>impressions that the log files are missing (and caching
>more generally)? Are the log tools being set up to screen
>too many pages out, or are they not logging everything?
>What do you think are the main causes of discrepancy
>between the 2 techniques? On e-consultancy we're currently
>running 2 measurement tools (Webtraffiq and LiveStats) -
>one client-side and one server-side - and the results are
>indeed often quite different. (separately we do our own
>tracking and analysis which we have much more faith in to
Director at WebtraffIQ
Agreed, it seems odd that there are so many web metrics firms pushing their service as the ‘most’ accurate, and those generated figures can indeed be very different.
I think that the main reason for this is due to commercial motivations. There are thousands of log file analysers in the UK and many more in the US. They are all competing for clients and a space in the market; some are part of ‘full service agency’ models of business and used as loss leaders, given free if a client takes on the companies core offering. (E.g. If the client agrees to taking on banner advertising, strategic link development, search engine optimisation and online PR -–the measurements of their efforts are thrown in free). As web metrics is not their core offering very often they use a much lesser product, such as a cheap logfile analyser and of course they will say its accurate!
Web site measurement originally came about as a by-product of web systems testing.
I think Jim Stern sums it up nicely…
“Log files are merely the result of good, solid engineering. If you create a system that does something, it's a good idea to record what happened. That way you've got something to look at when things go wrong.
Log files are, therefore, the results of a Web server doing its job, and not a formal effort to capture valuable business intelligence.” ( Source:- http://www.marketingprofs.com/3/sterne1.asp )
In answer to the question of ‘accurate’ technology for methods of data collection, we need only to look at the market leaders, specifically Webtrends.
Webtrends has gradually moved from logfile to browser based analysis. Webtrends to > Webtrends Live to > NetIQ. They have to some degree clouded the market perception of ‘accurate measurement’. When they moved from their original technology to ‘Webtrends Live’ they made many of their customers confused and angry. Therefore, they rebranded as NetIQ because they new they would be hemorrhaging clients and needed a Net to catch those clients. They also knew that they needed to appeal to the new ‘informed’ breed of client side marketers.
Hits v Page impressions --- Hits is a logfile term and as such is quite meaningless. Page Impressions are better but more useful to the Web Analysis Companies to gauge server usage and thus used as a metric for costing the service. Page impressions may go some way in gauging the accuracy and volume of data but I think it slightly misses the mark…
(Example... CPM was the key metric for gauging banner advertising, it was trendy and the industry standard in 1999. It wasn't until 2001 that companies realised that a click-through didn't amount to an actual acquisition. No wonder banner advertising fell dramatically during this period). First, companies need to take a holistic view of their goals and objectives then devise a single strategy for measurement.
I think a combination of technologies and measurement techniques to more accurately find a ‘unique user’ count & unique user action, will be the definitive Web/marketing metric. This will not only strengthen the web metrics Industry by being more understandable to the end user but will also allow above, below and through-the-line measurement harmonisation.
The fifth annual Online Measurement and Strategy Report, carried out in association with web analytics consultancy Lynchpin, contains a comprehensive analysis of issues affecting the web analytics industry and valuable insights into the use of analytics and business intelligence tools.
The Web Analytics Beginner's Guide aims to be the first step in a journey that will see you become a fully-fledged expert in online analytics. A must-read for those interested in this area, free to Bronze members and higher.
Free market research on digital marketing
Daily Pulse: award winning newsletter
It takes 30 seconds to register