Case study #1. Classic headline copy test

Headline copy tests are just as powerful for overlay A/B tests as they are for landing page tests. Plus, they are the easiest tests to run because you often don’t need to get your tech or design team’s help. 

Here’s one Reebok ran for its email opt-in overlay:

Version A                                              Version B

Reebok Version A Reebok Version B
Images from WhichTestWon, copyright protected.

Version B, with the benefit oriented JOIN AND SAVE! headline garnered 40% more email opt-ins than Version A.  Remember, this is a 40% lift in opt-ins from the exact same amount and type of traffic. 

If your email opt-in list could increase by 40% with a simple headline change, wouldn’t you test it now?    

Case study #2. Is bigger better for conversions?

An overlay greys-out (or whites-out) the page of the site it’s appearing on top of, so you can see a smidgen of the page below it, around the edges of the offer.  

The question for this test was, how big should your offer be? Generally online bigger is better after all….

Here’s the control overlay from hobby site FaveCrafts:

Favecraft Small
Image from WhichTestWon, copyright protected.

And here’s FaveCrafts’ test version where the offer box is even wider:

FavCraft Large
Image from WhichTestWon, copyright protected.

You’ll notice that both versions had the exact same copy and graphics… even the button size remained the same. The only change was the width of the offer box.  

So which one won?

The thinner one! Beating expectations, the thinner box pulled in 8.8% more email opt-ins than the wider one. 

Case study #3. How quickly should your overlay appear?

Many marketers personally find overlays annoying, so they want to delay them as long as possible into an average visitors’ arrival. 

The theory being that if you give visitors enough time to look around and fall in love with your site, they won’t mind the overlay as much.

Does it work? We ran this timing test on our own site.To put it in perspective, you should know our average new visit stays for 2.27 minutes. We split our new traffic into thirds and ran the exact same overlay to everyone…just at different times after arrival: 15-seconds, 30-seconds and 45 seconds.

Here’s what the overlay looked like (fairly bog standard): 

WTW Overlay Creative
Image from WhichTestWon, copyright protected.

And the winner was… 15 seconds!  Yes, we were surprised. 15 seconds doesn’t seem like a very long time for a visitor to decide if they like your site enough to sign up for your email newsletter.  Obviously our ‘marketers intuition’ was wrong so we’re glad we tested it.

The data: the 15-second timing beat 30-seconds by 11% and it beat 45-second timing by 50%.

So, if we’d gone with 45-second timing, we’d have 50% fewer newsletter readers.

Is faster overlay timing a best practice that will work for your site as well? Only testing can tell.

As the data shows, it’s well worth the test. And, luckily many email service providers (ESPs) make testing overlay timing fairly easy – it’s often functionality that’s included free with the service. But you have to use it.

Justin Rondeau is the producer of WhichTestWon’s The Live Event, an intensive two-day conference held each May featuring Case Studies and Workshops on how to optimize for better conversions.  Econsultancy readers can get $50 off their tickets by using the coupon code Econsultancy at