Could anybody provide me with some comparisons regarding the various packages on the market?

Mainly looking for comparisons in terms of;

ease of use
usefulness of stats provided
presentation of information
accuracy of statistics provided
the issue of ABC auditing
availabilty of realtime figures
associated costs

Also, interested to find out about monitoring products. Does anybody know about any reliable products that offer realtime monitoring of websites - availability testing/site down alerts/page download times/competitor comparisons etc.

Any information would be much appreciated.

This is the best package that I have ever used. Very comprehensive. Can generates reports in MSWord, HTML, RTF, etc. Includes graphs, tables, etc.

However, like all reporting software, you need to know how to interpret the results.

Ronnie

On 16:18:24 13 September 2001 grahamm wrote:
>Could anybody provide me with some comparisons regarding
>the various packages on the market?
>
>Mainly looking for comparisons in terms of;
>
>ease of use
>usefulness of stats provided
>presentation of information
>accuracy of statistics provided
>the issue of ABC auditing
>availabilty of realtime figures
>associated costs
>
>Also, interested to find out about monitoring products.
>Does anybody know about any reliable products that offer
>realtime monitoring of websites - availability
>testing/site down alerts/page download times/competitor
>comparisons etc.
>
>Any information would be much appreciated.

Hi Graham

There is actually a fair bit on this site already which might help you – especially in the forums and white papers sections. Try searches on ‘log files’, ‘clickstream’, ‘webtrends’ or similar and you should get some useful links back. There was a thread in this forum a month or two ago (scroll down this page) about methods of click stream analysis which discusses in quite some depth some of the “accuracy of statistics” issues you raise.

I know less about site monitoring products, though there is a company called SiteConfidence (www.siteconfidence.co.uk) which does what you say and does not cost too much (<£10k on the whole). They are relatively new to the market but seem to be successfully filling a need in that space. The limited experience I have had of them has been promising. At other ends of the monitoring spectrum you have everything from loads of freeware at one end up to the likes of Mercury Interactive whose enterprise-level products and services will set you back £100ks.

Of course, any serious e-commerce, CMS or eCRM application will give you advanced reporting and monitoring capabilities – at least for what it is doing. This would be the likes of Vignette, Broadvision, ATG, Broadbase, E.Piphany, eGain, MicroStrategy, Blue Martini, Net Perceptions, Interworld, Personify, Interwoven etc.. In many cases, if you have something like this in place you may not need a separate stats package.

If you are talking about companies that position themselves as pure reporting tools then the usual suspects would include: NetGenesis, Accrue, Engage, Webtrends, Mediahouse, RedEye, RedSheriff, Nedstat, Visual Insight, Macromedia. My impression is that Macromedia are moving back away from content management and eCRM type applications back to development tools so, personally, I would discount them (their product, which they acquired, is/was LikeMinds). The others mentioned you might be grouped as follows:

1. The bigger, more expensive, feature-rich, integrate-with-other-things, analytic platforms: NetGenesis, Accrue, Engage.
2. Log file analysis tools: Webtrends, Mediahouse (LiveStats)
3. Data/page tagging analysis tools: RedEye, RedSheriff, Nedstat, Visual Insight

*If you want to know the difference between 2 and 3 above then you should refer to the “click stream analysis” thread that has appeared in this forum before (search on ‘click stream’ to find it).

Briefly responding to your criteria headings:

>Ease of use
I am much more in favour of groups 1 and 3 above as they provide the means to create meaningful, marketing-friendly descriptions of data or data groups. The file names and query string paths that you tend to get from group 2 are often useless without more time-consuming analysis.

>Usefulness of stats provided
To some degree usefulness will be in the eye of the beholder – what is useful for some may not be so useful for others i.e. you should know what you want to know before you decide on a package. Despite the cleverness of many of these applications, in my experience they all need a fairly highly level of human intervention to configure and analyse the results they produce. Different people need different sorts of management information too. Senior management may just want a top level “management dashboard” type report, whereas the marketing people will want the detailed analysis of a particular marketing campaign.

That said, I would again favour groups 1 and 3 for usefulness of stats. Group 1 because of the integration with customer information that is possible and group 3 because its method and accuracy of click stream (path) analysis is superior to how group 2 does it. Otherwise, they all cover the same basic stats that you can get from log files, using cookies and Javascript i.e. page impressions, unique visitors, unique users sessions, page durations, session durations, visitor frequency, path analysis, user technologies, referring URLs and domains etc.

>Presentation of information
There are some differences in the interfaces and reporting modules (especially Visual Insight which, as its name suggests, tries to graphically represent how the site is being used), but not such that I have particularly noticed either way – they all do a pretty OK job.

>Accuracy of statistics provided

As I’ve said, I prefer the data/page tagging method. As with any data analysis job, the bottom line is ‘rubbish in, rubbish out’ i.e. it is not the reporting package that is likely to be at fault but the source data.

>The issue of ABC auditing

It looks like everyone who needs to go that way (big online publishers and software vendors) is going that way. A good thing for the credibility of the industry I guess. NB see article at end of this post for more on this subject.

>Availabilty of realtime figures

How real do you want your realtime ;) ? There’s real realtime and near realtime. A bit like Sky’s not-actually-on-demand Video/Films on demand. Doing real realtime takes a lot of processing power and that costs money. I’ve found that, actually, despite the thrill of true realtime, near realtime is perfectly good enough for most standard stats purposes– usually next day to allow processing to occur overnight. Where it is good to have realtime is when you’re running an online marketing campaign – especially e-mail or SMS – as the response times are in the 10 sec – 4 hour range. Here next day is not good enough. On the whole, though, you’ll probably be using a digital marketing/eCRM application that will do this reporting and analysis for you anyway.

>Associated costs

Very roughly… Group 1 will be upwards of £100k, Groups 2 and 3 are significantly less expensive ranging typically between £3k-£12k.

A few further points I would make on the subject:

1. Basic stats reporting (particularly of groups 2 and 3 above) is only a start. It tells you very little about actual customers – they exist only as IP addresses or cookies. Things get a lot more interesting and useful when you’re reporting on customer activity and propensity by segment or attribute. This is moving very much into the eCRM space which requires more customer focus and interaction, more advanced analytics etc. I have found that once you get your hands on customer-centric reporting the basic stats stuff becomes really very uninteresting – useful as a basic barometer but not much more.

2. Before worrying about stats packages, companies should spend more time thinking about *what* they want to know and why. They also need to think through who is going to do ongoing analysis and reporting. A Performance Measurement/ Management Information strategy is required.

3. A key criteria that you leave out that would strongly influence my decision making would be whether you are buying into a service-centric solution or a software-centric solution. For example, the likes of RedSheriff will be keen to offer services on top of their measurement technology, whereas Webtrends is a CD-ROM. I would think hard about the levels of support I would require as well – I would not want to have to rely on contacting the US during West Coast office hours, for example.

Ashley

P.S. Below is an article from netimperative, which although 6 months old now, points towards the increasing interest in measurement and ABC certification.

______________________________________________________
Measurement tops list of e-worries
by Hannah Sturgeon on 01 March 2001 08:00:00 GMT

More than one in three senior marketers are concerned about the credibility of online traffic statistics claimed by websites, according to a new survey.

As the industry’s attention turns to ensuring website readership figures are scrutinised to the same degree as those for offline media, the new report showed that senior marketers are expressing greater concern about the lack of accountability sites have for claimed user statistics.

The new report conducted at the Marketing Forum UK 2000 by measurement company ABC Electronic, showed that, compared to 1999, marketers are far less anxious about creativity and media buying effectiveness than about measurement.

Only 33% voiced concern about online creativity, compared to 43% in 1999, while nearly 40% voiced measurement as a major concern compared to 31% last year.

More significantly, the report showed that 24% ‘strongly agreed’ and 62% ‘agreed’ that third party internet audits and certification are crucial to the media buying decision. Furthermore 33% ‘strongly agreed’ and 55% ‘agreed’ that there should be one industry standard for the audit of sites.

In a statement, ABC Electronic, which aims to be recognised as that one official standard, said: “These findings support the UK media industry's work over the last four years, resulting in an industry agreed and managed set of metrics being used. Independently-audited data is essential in understanding the growth of the internet in the UK, Ireland and globally.”

The issue of costs to small businesses of gaining official accreditation, however, is still one major hurdle of agreeing to new measurement standards.

The results of the survey come after the high-profile sacking of e-district.net CEO Stephen Laitman who was accused of “substantially overstating” site user statistics. The Serious Fraud Office is investigating Laitman on behalf of e-district.

Mark Terry communications director at Rivals.net, which is campaigning for the adoption of a single standard measurement service, said: “These results come as no surprise to us; in fact I thought they would be far higher.

“We are trying to push for accountability; we want brands to boycott those sites that do not have a regular accredited way of being audited. It should be part of a company’s cost of sales.”

Terry said that websites should have a quarterly measurement system similar to Barb for TV ratings and ABC measurements for newspapers, magazines and radio.

“Brands should be most suspicious about what they are getting if a site does not have recent ABC metrics; it is one of the first questions we are asked,” he said.
______________________________________________________

On 16:18:24 13 September 2001 grahamm wrote:
>Could anybody provide me with some comparisons regarding
>the various packages on the market?
>
>Mainly looking for comparisons in terms of;
>
>ease of use
>usefulness of stats provided
>presentation of information
>accuracy of statistics provided
>the issue of ABC auditing
>availabilty of realtime figures
>associated costs
>
>Also, interested to find out about monitoring products.
>Does anybody know about any reliable products that offer
>realtime monitoring of websites - availability
>testing/site down alerts/page download times/competitor
>comparisons etc.
>
>Any information would be much appreciated.

I was pitched an interesting web analysis tool this week. It is called www.hitwise.com . They buy log files from a selection of ISP's and then analayse the data. It seems a very good way to get a snapshot of your own and other people's business.

You can see where your traffic comes from.... and where it then goes to.... And the same for your competitor's sites. You can also see what proportion of the online traffic your site is attracting; this info can be segmented into market categories, the UK universe and the "all site" universe. It's very interesting.

Barney

On 11:56:39 4 October 2001 Ashley wrote:
>Hi Graham
>
>There is actually a fair bit on this site already which
>might help you – especially in the forums and white
>papers sections. Try searches on ‘log files’,
>‘clickstream’, ‘webtrends’ or
>similar and you should get some useful links back. There
>was a thread in this forum a month or two ago (scroll down
>this page) about methods of click stream analysis which
>discusses in quite some depth some of the “accuracy
>of statistics” issues you raise.
>
>I know less about site monitoring products, though there
>is a company called SiteConfidence
>(www.siteconfidence.co.uk) which does what you say and
>does not cost too much (<£10k on the whole). They
>are relatively new to the market but seem to be
>successfully filling a need in that space. The limited
>experience I have had of them has been promising. At other
>ends of the monitoring spectrum you have everything from
>loads of freeware at one end up to the likes of Mercury
>Interactive whose enterprise-level products and services
>will set you back £100ks.
>
>Of course, any serious e-commerce, CMS or eCRM application
>will give you advanced reporting and monitoring
>capabilities – at least for what it is doing. This
>would be the likes of Vignette, Broadvision, ATG,
>Broadbase, E.Piphany, eGain, MicroStrategy, Blue Martini,
>Net Perceptions, Interworld, Personify, Interwoven etc..
>In many cases, if you have something like this in place
>you may not need a separate stats package.
>
>If you are talking about companies that position
>themselves as pure reporting tools then the usual suspects
>would include: NetGenesis, Accrue, Engage, Webtrends,
>Mediahouse, RedEye, RedSheriff, Nedstat, Visual Insight,
>Macromedia. My impression is that Macromedia are moving
>back away from content management and eCRM type
>applications back to development tools so, personally, I
>would discount them (their product, which they acquired,
>is/was LikeMinds). The others mentioned you might be
>grouped as follows:
>
>1. The bigger, more expensive, feature-rich,
>integrate-with-other-things, analytic platforms:
>NetGenesis, Accrue, Engage.
>2. Log file analysis tools: Webtrends, Mediahouse
>(LiveStats)
>3. Data/page tagging analysis tools: RedEye, RedSheriff,
>Nedstat, Visual Insight
>
>*If you want to know the difference between 2 and 3 above
>then you should refer to the “click stream
>analysis” thread that has appeared in this forum
>before (search on ‘click stream’ to find it).
>
>Briefly responding to your criteria headings:
>
>>Ease of use
>I am much more in favour of groups 1 and 3 above as they
>provide the means to create meaningful, marketing-friendly
>descriptions of data or data groups. The file names and
>query string paths that you tend to get from group 2 are
>often useless without more time-consuming analysis.
>
>>Usefulness of stats provided
>To some degree usefulness will be in the eye of the
>beholder – what is useful for some may not be so
>useful for others i.e. you should know what you want to
>know before you decide on a package. Despite the
>cleverness of many of these applications, in my experience
>they all need a fairly highly level of human intervention
>to configure and analyse the results they produce.
>Different people need different sorts of management
>information too. Senior management may just want a top
>level “management dashboard” type report,
>whereas the marketing people will want the detailed
>analysis of a particular marketing campaign.
>
>That said, I would again favour groups 1 and 3 for
>usefulness of stats. Group 1 because of the integration
>with customer information that is possible and group 3
>because its method and accuracy of click stream (path)
>analysis is superior to how group 2 does it. Otherwise,
>they all cover the same basic stats that you can get from
>log files, using cookies and Javascript i.e. page
>impressions, unique visitors, unique users sessions, page
>durations, session durations, visitor frequency, path
>analysis, user technologies, referring URLs and domains
>etc.
>
>>Presentation of information
>There are some differences in the interfaces and reporting
>modules (especially Visual Insight which, as its name
>suggests, tries to graphically represent how the site is
>being used), but not such that I have particularly noticed
>either way – they all do a pretty OK job.
>
>>Accuracy of statistics provided
>
>As I’ve said, I prefer the data/page tagging method.
>As with any data analysis job, the bottom line is
>‘rubbish in, rubbish out’ i.e. it is not the
>reporting package that is likely to be at fault but the
>source data.
>
>>The issue of ABC auditing
>
>It looks like everyone who needs to go that way (big
>online publishers and software vendors) is going that way.
>A good thing for the credibility of the industry I guess.
>NB see article at end of this post for more on this
>subject.
>
>>Availabilty of realtime figures
>
>How real do you want your realtime ;) ? There’s real
>realtime and near realtime. A bit like Sky’s
>not-actually-on-demand Video/Films on demand. Doing real
>realtime takes a lot of processing power and that costs
>money. I’ve found that, actually, despite the thrill
>of true realtime, near realtime is perfectly good enough
>for most standard stats purposes– usually next day
>to allow processing to occur overnight. Where it is good
>to have realtime is when you’re running an online
>marketing campaign – especially e-mail or SMS
>– as the response times are in the 10 sec – 4
>hour range. Here next day is not good enough. On the
>whole, though, you’ll probably be using a digital
>marketing/eCRM application that will do this reporting and
>analysis for you anyway.
>
>>Associated costs
>
>Very roughly… Group 1 will be upwards of
>£100k, Groups 2 and 3 are significantly less
>expensive ranging typically between £3k-£12k.
>
>
>A few further points I would make on the subject:
>
>1. Basic stats reporting (particularly of groups 2 and 3
>above) is only a start. It tells you very little about
>actual customers – they exist only as IP addresses
>or cookies. Things get a lot more interesting and useful
>when you’re reporting on customer activity and
>propensity by segment or attribute. This is moving very
>much into the eCRM space which requires more customer
>focus and interaction, more advanced analytics etc. I have
>found that once you get your hands on customer-centric
>reporting the basic stats stuff becomes really very
>uninteresting – useful as a basic barometer but not
>much more.
>
>2. Before worrying about stats packages, companies should
>spend more time thinking about *what* they want to know
>and why. They also need to think through who is going to
>do ongoing analysis and reporting. A Performance
>Measurement/ Management Information strategy is required.
>
>3. A key criteria that you leave out that would strongly
>influence my decision making would be whether you are
>buying into a service-centric solution or a
>software-centric solution. For example, the likes of
>RedSheriff will be keen to offer services on top of their
>measurement technology, whereas Webtrends is a CD-ROM. I
>would think hard about the levels of support I would
>require as well – I would not want to have to rely
>on contacting the US during West Coast office hours, for
>example.
>
>Ashley
>
>P.S. Below is an article from netimperative, which
>although 6 months old now, points towards the increasing
>interest in measurement and ABC certification.
>
>______________________________________________________
>Measurement tops list of e-worries
>by Hannah Sturgeon on 01 March 2001 08:00:00 GMT
>
>More than one in three senior marketers are concerned
>about the credibility of online traffic statistics claimed
>by websites, according to a new survey.
>
>As the industry’s attention turns to ensuring
>website readership figures are scrutinised to the same
>degree as those for offline media, the new report showed
>that senior marketers are expressing greater concern about
>the lack of accountability sites have for claimed user
>statistics.
>
>The new report conducted at the Marketing Forum UK 2000 by
>measurement company ABC Electronic, showed that, compared
>to 1999, marketers are far less anxious about creativity
>and media buying effectiveness than about measurement.
>
>Only 33% voiced concern about online creativity, compared
>to 43% in 1999, while nearly 40% voiced measurement as a
>major concern compared to 31% last year.
>
>More significantly, the report showed that 24%
>‘strongly agreed’ and 62% ‘agreed’
>that third party internet audits and certification are
>crucial to the media buying decision. Furthermore 33%
>‘strongly agreed’ and 55% ‘agreed’
>that there should be one industry standard for the audit
>of sites.
>
>In a statement, ABC Electronic, which aims to be
>recognised as that one official standard, said:
>“These findings support the UK media industry's work
>over the last four years, resulting in an industry agreed
>and managed set of metrics being used.
>Independently-audited data is essential in understanding
>the growth of the internet in the UK, Ireland and
>globally.”
>
>The issue of costs to small businesses of gaining official
>accreditation, however, is still one major hurdle of
>agreeing to new measurement standards.
>
>The results of the survey come after the high-profile
>sacking of e-district.net CEO Stephen Laitman who was
>accused of “substantially overstating” site
>user statistics. The Serious Fraud Office is investigating
>Laitman on behalf of e-district.
>
>Mark Terry communications director at Rivals.net, which is
>campaigning for the adoption of a single standard
>measurement service, said: “These results come as no
>surprise to us; in fact I thought they would be far
>higher.
>
>“We are trying to push for accountability; we want
>brands to boycott those sites that do not have a regular
>accredited way of being audited. It should be part of a
>company’s cost of sales.”
>
>Terry said that websites should have a quarterly
>measurement system similar to Barb for TV ratings and ABC
>measurements for newspapers, magazines and radio.
>
>“Brands should be most suspicious about what they
>are getting if a site does not have recent ABC metrics; it
>is one of the first questions we are asked,” he
>said.
>______________________________________________________
>
>
>
>On 16:18:24 13 September 2001 grahamm wrote:
>>Could anybody provide me with some comparisons
>regarding
>>the various packages on the market?
>>
>>Mainly looking for comparisons in terms of;
>>
>>ease of use
>>usefulness of stats provided
>>presentation of information
>>accuracy of statistics provided
>>the issue of ABC auditing
>>availabilty of realtime figures
>>associated costs
>>
>>Also, interested to find out about monitoring
>products.
>>Does anybody know about any reliable products that
>offer
>>realtime monitoring of websites - availability
>>testing/site down alerts/page download
>times/competitor
>>comparisons etc.
>>
>>Any information would be much appreciated.

All traffic analysis software use one of two approaches: (1) log file analysis and (2) clickstream analysis using cookies.

Of (1), Webtrends is the most well known, but thousands of webmasters still swear by the free Analog
http://www.statslab.cam.ac.uk/~sret1/analog/
The weaknesses of log analysis are well explained here
http://www.arsdigita.com/books/panda/user-tracking
and
http://www.acm.org/archives/wa.cgi?A2=ind0009E&L=chi-web&P=R1083

(2) This service is usually provided by dedicated 3rd party data warehousing companies, like Red Sherrif or Redeye. The principle is explained here
http://www.arsdigita.com/asj/clickstream/
and
http://www.acm.org/archives/wa.cgi?A2=ind0010A&L=chi-web&P=R6865

There are privacy issues with the latter approach, also explained here
http://www.arsdigita.com/books/panda/user-tracking

The CHI-WEB mailing list has often discussed user tracking. I reference a few above, but you can find all messages on the subject here:
http://www.acm.org/archives/wa.cgi?S2=chi-web&q=&s=log+analysis&f=&a=&b=

francois