When your sales conversion is lower than expected - asking the tech team for web performance metrics may not help: visitors per hour is not enough, concurrent users is plain unhelpful.
I was interested to read Ashley's blog, where he argued that marketing and commercial people "should take a closer look at their IT set up?"
Again this week, I came across the sad case of a marketing manager trying to get the best ROI on their portal, and getting baffled by the wrong metrics.
I guess I'm surprised that after all these years (back in 1992 I was explaining the benefits of the web to blue chips), that marketing and business folk still find it hard to get concrete measurements, so they can apply good business sense to the online world.
Sales conversions had been lower than expected and the question was raised - just how many sales /minute can our online store handle?
The web analytics in place, with a little more time, could have provided more than just pages/minute and visitors/minute metrics.
Pages or visitors per minute is a poor measure anyway your site can probably shift a lot more homepages per second, than say the final checkout page (which does a load of database and credit checking work), and not all visitors are on your making-money pages anyway.
The finance metrics in place weren't bad either, that told them how many orders per day they'd taken. But that was all historic data about the 'what happened', not 'why did it happen'.
It couldn't help the marketing manager's question - the successful campaign had generated lots of new visitors, but had that extra load maybe caused caused slow downs and errors for users; and thus been the root cause of the low sales conversions?
Bluntly, did the online technology let the campaign down just at it's moment of success. Or must root cause lie in the campaign marketing and targeting.
The poor marketing manager was left high and dry when the tech team responded to the web capacity query with a big number.
It would have been nice if they could have shown a graph like the following of performance for the checkout pages:
Users are treated worse as the week goes on... and they said this portal could handle the best part of a thousand 'concurrent users'. Sounds huge. Humm..
But err… not all 'concurrent users' are making purchases… so what exactly does it mean?
The business and tech team do talk a different language - the tech team are wrestling with the vagaries and not-so-logical problems and black-art of complex web application software and languages, while the marketers are coming top down, from customer user journeys, and sales values, etc.
The tech team uses concurrent users as a measure for a simple reason. It's easy. It requires no testing. It requires no analysis of any data. No need to spend any time on it. A web application worth its salt will give you a value for it, straight off.
And for a busy site, it is often a reassuring high value - way bigger than any business metrics like sales per minute. It kind of deflects criticism of the technology platform.
Definiton: Concurrent Users is the count of how many real users are currently on the site (in technical talk - they have a session running).
- When does a user become a Concurrent User? - as soon as they hit their first page.
- When does a user drop out of the count? That happens after a timeout value is reached, after the last page they requested. The length of that timeout is configurable in the web application - e.g. 30 minutes.
So not all concurrent users are equal - they may be a visitor who has grabbed one page in the last half hour; or at the other end of the scale, someone who went through 20 pages in the last 10 minutes, during which they searched for a product, put it in the basket, then went to the check out to buy it.
That's not a great metric for the marketer struggling with low sales conversion issues.
And another factor - is that by changing the configuration of your web software (the session timeout value), the concurrent users value will also change! Without any change in the level of activity on your system. Or any change in the capacity of your system to handle load. Ouch.
So what have we got so far:
- Pages per second is a poor measure of capacity - your site can probably shift a lot more homepages per second, than say the final checkout page (which does a load of database and credit checking work)
- Concurrent users are far from being equal - ranging from a visitor who requests only 1 page in say 30 minutes - versus a valuable customer who in 15 minutes gets 10 pages, makes a product search and spends money with you.
- The concurrent user number from your web application can be manipulated up or down without any change to your systems capacity, through the time-out setting.
There are of course better metrics to cover the capacity handling capability of your website, which I'll cover next time.
But until then, if you're planning some load testing of your website - make sre that you'll get more then 'concurrent users' out of it.
Deri Jones is CEO and founder of SciVisum, a web application testing company.