Craig Sullivan is Group eBusiness Customer Experience Manager at Belron, a company which includes Autoglass among its brands. 

Craig spoke at the first JUMP event last year, on the subject of using online data to optimise offline marketing.  

In this interview, taken from our recent JUMP magazine, he talks about using multivariate testing to improve the effectiveness of TV ads, as well as online usability...

Tell me about your JUMP presentation...

Online is still too often thought of as a separate, hermetically sealed bubble, set apart from the rest of the business. This is something that is really dumb, and it makes me angry. 

Online is measurable, you can be experimental; you can test different things and find out a lot that you couldn’t possibly find out elsewhere. 

The important thing to realise is that it isn’t just a channel that operates alone, the journey for the customer doesn’t end when they press the submit button on the website. 

There are three parts of the triangle; 

1. Having the technical capability to provide the experience for users.

2. Having the business capability.

3. The user’s experience of all of this. .

Some online people can be too much like Pontius Pilate, dealing with what happens online in their area of responsibility, then washing their hands and not caring what happens to customers after that. 

You have to have an eye on what customers are doing before and after they visit the website; the SEO copy they are seeing, the TV and print advertising they read, all of this is important. 

After all, you can build the world’s greatest online experience, but if the call centre or store staff screw it up by giving the customer poor service, what’s the point? 

How have you used online data to improve offline marketing? 

This Monday, we’ll be launching our first TV ad starring Izzie, a female technician. Last year, we had Gavin on the ads, and he has a large Facebook following, and has been popular with customers. 

However, using multivariate testing (MVT) I was keen to test what works differently on the site when our TV ads are on, and when they’re off air. 

We tested images of several different people on the website, including Gavin from the TV as, and I included Izzie to see what effect this would have. 

When we had the same person on the site as on the TV ad, this worked well, but we had a huge uplift when the female technician was shown on the site at the same time as TV ads were showing. 

The reason we found was that these TV adverts were often shown in daytime TV slots. It wasn’t that this audience was predominantly female, but we found that the people who were engaged enough to act on the ad and visit the site were female. 

So this told us that we didn’t always have to have the same person on the website and the TV ad, the reaction may be different depending on the kind of person that heads to the site. 

Using Izzie led to a 10% increase in conversions, so the obvious question was: why not use Izzie in the TV ads? 

This is one way in which we have used online data offline, but this data can also feed into other areas. 

For example, we can take some of the people scheduled to appear in TV ads and test them on the website first for strong reactions from users; positive or negative. In this way online testing is helping offline advertising. 

Online testing drives insight, and can drive changes both on and offline. 

How much data do you have to ‘play with’? 

We’re testing over a million different recipes each month, buttons, copy, imagery of vans, and more, building up a library of what works, the audience it works on, and once we have a well baked recipe, we can use it again, confident that it has been proven to work. 

For example, I know that certain images work best when users have typed in the term ‘windscreen repair’ on a search engine before arriving at the site, so that user sees a carefully tested version of the landing page that has been proven to work. 

Is it a challenge to manage such a variety of data and turn in into actionable insights? 

Everyone has too much data and not enough insight, and what is important is to use the data to produce actionable insights. 

A lot of people, I call them report monkeys, are just churning out reports, but you need to get into the mindset of the customer to have a better understanding of the data and how it can be used to improve the business. 

This is where looking across channels is so valuable; you have to do an analysis at local level to get an understanding of what works and where. 

Some people in my position are drowning in a sea of data, nit insight. For us it’s about producing models that people can use to tune their local offering. There is a great deal that we can standardise though which helps. 

Can you measure customers across channels? 

We can use dynamic phone numbers, so we can tell which point the customer was at in the purchase funnel before they made a call. 

What people also don’t do is look at the effect that online changes have offline. So, 5% uplift on conversions is great for the online side of the business, but the change that cause it may also have resulted in dropped call volumes. 

You have to measure the flow of data to other channels, looking at where you can make changes that are good for the business as a whole. 

One classic multichannel example is when one of our businesses came to us, as they were worried that a campaign they were running would produce an uplift in customer calls that they wouldn’t be able to handle. 

So we changed the function on the website so they contact number wasn’t shown, and instead produced an email contact option. 

This bought time for the call centre to clear the backlog, and is one example of how the online channel can support the rest of the business. 

Once you want to act on what you have learned from testing data, are there challenges in getting the rest of business to learn the lessons and take action? 

I’m lucky that my company is willing to learn from online data and use it to improve other areas of the business, and this is what a good multichannel business should do. 

My job online takes me out into other areas of the business where other online people would not normally venture, and this is because you need to be customer focused and see the customer journey beyond the website. 

You have to care about what happens beyond your own area of the business. 

Graham Charlton

Published 27 June, 2011 by Graham Charlton

Graham Charlton is editor in chief at SaleCycle, and former editor at Econsultancy. Follow him on Twitter or connect via Linkedin.

2566 more posts from this author

You might be interested in

Comments (3)

Will Quick

Will Quick, Media Planner at MediaCom

Wow, awesome work, Craig. Taking an almost holistic approach to conversion testing is really inspired. I can't begin to imagine the processes you have in place to accurately map all of these funnel entrances and conversions, but I'm damn well going to think about it and try to apply this!

Really, really great.

- Will

about 7 years ago


Craig Sullivan, Customer Experience Manager at Belron International

Will, thank you.

Actually setting up this stuff is fairly easy. The hard bit is working on the freaking question you want to ask!

First, work back from the outcome - what do you want to happen? What's insightful? What questions are you asking? What happens if it doesn't go your way? Is that the right question? Can I even measure that stuff?

OK - so let's say I'm thinking something like "Does chat improve conversion". I think - ok, that's interesting but how about

"Does chat improve conversion and how does it change other contact channels" - better.

Then a slight tweak

"What pages does offering chat act to improve or reduce conversion across a weekly cycle and how does this change contact volumes across all channels" - that's good.

Now, all I need is to make sure that this lot, are *instrumented* to help me answer that question:

Optimost console - for the A/B testing
Google analytics - for segmentation/checking
Clicktale - for analysing dropoff areas, getting insight
Browsercam - for browser testing and checking any bugs/issues

This stuff probably took about 2-3 days to get all tweaked and configured but it was worth it. We added event tracking to Google analytics and then set up some unique phone numbers. It wasn't taxing for the developers - just fiddly and boring.

It's hard to ask the 'right' questions but the one thing I still don't always get right is the instrumentation.

If you can't measure it, your test just became the biggest air guitar in the world.

Will - I have a document about 'holistic site health' that you might find interesting. Mail me direct for a copy or via Linkedin.


about 7 years ago


Depesh Mandalia, CEO & Founder at SM Commerce

This is a fantastic example of a joined up approach - you mentioned "I’m lucky that my company is willing to learn from online data and use it to improve other areas of the business, and this is what a good multichannel business should do." - this is not always easy in practice as testing can involve many stakeholders and many differing agendas (should i invest time in supporting this VS other direct trade activities for example) - especially getting your Marketing team to push a different ad based on online data.

The turnaround time is quite impressive too - worth noting that spending time refining the question (as you've done) makes a huge difference in what you deliver via MVT and how you measure it against what is important to your business

about 7 years ago

Save or Cancel

Enjoying this article?

Get more just like this, delivered to your inbox.

Keep up to date with the latest analysis, inspiration and learning from the Econsultancy blog with our free Digital Pulse newsletter. You will receive a hand-picked digest of the latest and greatest articles, as well as snippets of new market data, best practice guides and trends research.