I'd second that Ashley - and not just because my e-mail response is the featured article in this month's Attuned!

You are correct that quality resources dedicated to site measurement are in short supply. Readers would do well to visit emetrics.org, which is the result of last year's inaugural E-metrics Summit in Santa Barbara.

Attendees expressed a collective desire to "justify Web expenditures and quantify the results", which, as we all know, is a walk in the park ;-)

David Brown
Clickstream Technologies

Hi Ashley,

Get in touch whenever it is convenient.

I followed up my e-mail to Hurol with another, which explains the frustration of client-side tracking vendors such as Clickstream.

We pick up a client, previously using a server log analysis product and start producing Page Impression figures anything from 20% to 80% higher than those reported by the log analysis tool. The usurped vendors are ABCe associates - as are we - and the client cannot understand how the previous figures and ours can both be 'right'. We explain the principle of auditing and the fact that it is the tool and the methodology rather than the figures themselves which are audited, but this rarely cuts any ice with the client.

The result is the client having to explain to their advertisers why they are about to increase their placement charges and, when they next audit, having to explain to their market the sudden leap in online traffic.

There is potential here for the client to lose faith in the auditing process in particular and online measurement in general and, indeed, we have seen this happen on more than one occasion.

The client might be happy with us, but believe the whole web measurement area lacks credibility. So long as we in the analytics business are telling companies that they had 5 million page impressions last month...but that they also had 5.1,5.2....6, 7, 8 million, depending on the tool selected, I believe this situation can only deteriorate.

As a former employee of a log analysis tool vendor who has moved to what I consider best-of-breed, client-side tracking, I accept that I am somewhat less than objective when it comes the merits/demerits of each methodology :-). Leaving aside the question of which is superior, everybody in this space should at least be able to agree that we work towards some form of consistency for measuring headline metrics such as page impressions.

Making analytics work for any online business is not and never will be an exact science. However, with one or two caveats, the calculation of core metrics like page impressions can be. The better job everyone in the industry can do to convince end-users that their figures are genuine and credible, the easier it should be to encourage the same people to use these numbers as the foundation for extended metrics, ROI calculation and the like…pushing the envelope of web analytics in the way we all want to see this industry develop.

The logical first step in this process is to ensure we vendors are all singing from the same hymn sheet when it comes to first principles of how to measure simple metrics. How long can the industry sustain the notion that widely divergent figures produced by competing tools are both 'right'? Moreover, will organisations continue to accept such deviations in measurement indefinitely?

Regards,

David Brown.

On 16:58:58 14 January 2003 Ashley wrote:
>Hi David
>
>My book was featured (and glowingly reviewed) in the
>latest 'Attuned' so I too have some vested interest to
>declare although I don't think you can access the archives
>so any new sign-ups won't benefit me...
>
>We're working on a white paper/report on web metrics and
>will be featuring selected vendors. It will be out around
>March and available to our 8,600+ registered users. If you
>don't mind I'll contact you separately about this as
>Clickstream is on our list of vendors to feature.
>
>Ashley

Hi Ashley,

It's a combination of factors:

1 - Web server log files will fail to record to any content served by caching servers or proxies (one or two of the better tools can accommodate some proxy logs).

2 - They also fail to spot pages served from the local browser cache. This is increasingly THE problem as browser timeouts increase with every new version of IE.

3 - By definition, a log file writes a sequential record for each request made - meaning there are no two-point page timings.

4 - Although many of the better log file analysis tools will filter out 'noise' at point of database ingestion, the fact that every request for every resource is initially recorded, creates its own problems in terms of size, file transfer, sewing, and the general unwieldiness of it all.

5 - Users are identified courtesy of their IP address (or IP address plus browser version). The AOL factor, increasing use of proxy servers in general, dynamic IP addresses, etc., all ensure that this is wildly inaccurate. Again, some of the better log file tools can incorporate use of cookies into the unique user identification process, but if a site is not deploying cookies, the best they can do for is IP address.

I used to work for a well-known log analysis vendor and I know that they now include a tagging option to enhance the log file data. Great, but this simply demonstrates the inherent inaccuracy of log file tools.

As I said, even on dynamic sites, we have witnessed 20% caching of pages. On static sites, this figure has been considerably higher. Further, as a general rule of thumb, the more popular the page, the more likely it is to be cached (either by the site owner to enhance the visitor's experience, or by the browser as the page has been previously viewed). So it's not simply a question of the quantity of pages missed by log file tools, but the quality. An unusually low 5% caching might not be considered material until you discover that missing this 5% completely distorts your impression of the most popular pages on your site, perhaps rendering some of your most popular content some of the least.

Again, the most popular pages are often those that enjoy the longest page view durations (not always, but sometimes). So 5% caching of pages might translate to 25% of total time on site. We have bona fide experience of a client whose page caching was 40%, but these pages accounted for 80% of the total time visitors spent on site. A log file tool, whilst missing 40% of page views (which is bad enough), would have misappropriated 80% of page view duration to other pages.

Thus, the inability of log file tools to record cached content is a double-edged sword. Not only is there a shortfall in data, but the data recorded can be skewed out of all resemblance to reality.

If users of log file tools think they can simply up all their metrics by 20% (or whatever their techies are telling them is the incidence of caching on their site), then they are kidding themselves.

Regards,

David Brown.

On 11:06:49 15 January 2003 Ashley wrote:
>David
>
>I agree (and applaud) what you say. It certainly has to
>happen, and market forces will inevitably ensure it does.
>Indeed, this topic was the cover story on NMA this week.
>
>In theory I believe client-side measurement must be more
>accurate as there is so much less 'noise' created by the
>redundant data included in log files. However, I'm
>interested that you have found your client-side
>measurement to *increase* page impressions by so much -
>why do you think this is? Is it 'back' click page
>impressions that the log files are missing (and caching
>more generally)? Are the log tools being set up to screen
>too many pages out, or are they not logging everything?
>
>What do you think are the main causes of discrepancy
>between the 2 techniques? On e-consultancy we're currently
>running 2 measurement tools (Webtraffiq and LiveStats) -
>one client-side and one server-side - and the results are
>indeed often quite different. (separately we do our own
>tracking and analysis which we have much more faith in to
>be honest)