How long in term of visits or conversion do you need to run an A/B test before you know the result of the test.

Hi Denis

Thank you for the response.

I thought there would be a specific calculation that could be done to determine how many visits or conversions were required to get a statistically significant result.

Michael

hi, Michael, if you're worried about statistical significance, a simple technique is to run an A/B/A test. ie: run the split 3 ways, but keep 2 of those groups identical (call them 'A' & 'A2').

Here're 2 examples to show how this helps you:

Test 1:

• group splits: 33% A, 33% B, 33% A2.
• results: 200 sales A, 250 sales B, 201 sales A2

0.5% variance between the two 'A' groups gives you confidence

Test 2:

• group splits: 33% A, 33% B, 33% A2.
• results: 200 sales A, 250 sales B, 250 sales A2.

25% variance between the two 'A' groups means the statistical significance of the test isn't great. On top of that, the difference between the 'A' & 'A2' groups is equal to the difference between A/B. You can't safely make any conclusions based on this result & need to either run the test for longer or rethink it.

Does this help?

daniel

Duration is entirely dependant on what you are testing. 

An example?  Email landing pages may only need to test for a few hours (as you are driving considerable traffic).  Testing the color, size, shape, position, text of an add to basket button....the same thing.

If you are testing very specfic or detailed things...such as copy on your SEO/SEM landing pages by Time of Day; by product/key word...you'll need to test longer for statically relevant results.

Also consider multi-step process...such a persona modeling - you might start with data application (overtly collected/preference), then you have to wait for a user to come back to the website to test interaction with targeted product or messaging.   

DANIEL B - GREAT insight on the A/B/A - sounds like you've done this before!

REMEMBER - Google site optimizers CORE intent is page content optimization and testing - not process testing or user experience testing ( that invloved more than a page or two, larger "cross session" experience experience testing, segement/persona based optimization. 

It can be "made" to do all of the above (love the google gods), but the "duration calculator" feels it's more oriented to simple tests.

On 14:35:06 6 August 2008 tmguys wrote:

You can use this handy calculator to identify how long you need to run the test to acheive an appropriate confidence level which is the best design:

https://www.google.com/analytics/siteopt/siteopt/help/calculator.html