Showing posts 1 - 10 of 13
1. Director at Quayside Clothing Limited

05 August 2008 13:06pm

How long in term of visits or conversion do you need to run an A/B test before you know the result of the test.

2. Technical Project Manager (MBA, MBCS, CITP, CEng) at Naxtech.com

05 August 2008 14:11pm

Hi,

There are no rules written in stone about this.  Some choose to do it for 5% of their total monthly traffic or visitors.  Others base it on a number of days.

Given that you know your customers' needs and practices...just base it on what feels right to you, for your business.

Does that help at all?

regards,

Denis
www.naxtech.com

3. Director at Quayside Clothing Limited

05 August 2008 14:39pm

Hi Denis

Thank you for the response.

I thought there would be a specific calculation that could be done to determine how many visits or conversions were required to get a statistically significant result.

Michael

4. Director at Quayside Clothing Limited

05 August 2008 14:39pm

Hi Denis

Thank you for the response.

I thought there would be a specific calculation that could be done to determine how many visits or conversions were required to get a statistically significant result.

Michael

5. E-Business Consultant at Dan Barker

05 August 2008 15:16pm

hi, Michael, if you're worried about statistical significance, a simple technique is to run an A/B/A test. ie: run the split 3 ways, but keep 2 of those groups identical (call them 'A' & 'A2').

Here're 2 examples to show how this helps you:

Test 1:

• group splits: 33% A, 33% B, 33% A2.
• results: 200 sales A, 250 sales B, 201 sales A2

0.5% variance between the two 'A' groups gives you confidence

Test 2:

• group splits: 33% A, 33% B, 33% A2.
• results: 200 sales A, 250 sales B, 250 sales A2.

25% variance between the two 'A' groups means the statistical significance of the test isn't great. On top of that, the difference between the 'A' & 'A2' groups is equal to the difference between A/B. You can't safely make any conclusions based on this result & need to either run the test for longer or rethink it.

Does this help?

daniel

6. Director at Quayside Clothing Limited

05 August 2008 15:28pm

Hi Daniel

Thank you for the information.

I wasn't aware of A/B/A testing. Is this commonly used ?

Michael

7. Founder at LightsOut Marketing, LLC

05 August 2008 19:25pm

Duration is entirely dependant on what you are testing.

An example?  Email landing pages may only need to test for a few hours (as you are driving considerable traffic).  Testing the color, size, shape, position, text of an add to basket button....the same thing.

If you are testing very specfic or detailed things...such as copy on your SEO/SEM landing pages by Time of Day; by product/key word...you'll need to test longer for statically relevant results.

Also consider multi-step process...such a persona modeling - you might start with data application (overtly collected/preference), then you have to wait for a user to come back to the website to test interaction with targeted product or messaging.

DANIEL B - GREAT insight on the A/B/A - sounds like you've done this before!

06 August 2008 14:35pm

You can use this handy calculator to identify how long you need to run the test to acheive an appropriate confidence level which is the best design:

9. Founder at LightsOut Marketing, LLC

06 August 2008 15:04pm

REMEMBER - Google site optimizers CORE intent is page content optimization and testing - not process testing or user experience testing ( that invloved more than a page or two, larger "cross session" experience experience testing, segement/persona based optimization.

It can be "made" to do all of the above (love the google gods), but the "duration calculator" feels it's more oriented to simple tests.

On 14:35:06 6 August 2008 tmguys wrote:

You can use this handy calculator to identify how long you need to run the test to acheive an appropriate confidence level which is the best design:

10. Enterprise

Founding Partner & CEO at Essence

06 August 2008 16:08pm

Hi Michael,

All posts above have sensible advice.  I'm not going to labour the same points but did just want to point out that Google's calculator is an 'estimator' and is designed to help you predict how long a test needs to run for prior to running it.  Worth saying that all of Dennis'  comments earlier are along the same lines - you can't calculate statistical confidence until you have actual test results to go on so by definition anything using number of days, or % of visitors etc. is a 'rule of thumb'

Once your test is live you can use a real stats test to determine the statistical significance.  That means you can keep plugging your test results into the tests stats until you reach your desired confidence level.  Sadly it is not trivial and I can't point you to an online calculator.  But I do have a spreadsheet which does the calculation.  Feel free to contact me for a copy.

Incidentally, A/B/A' is indeed a very powerful method for ensuring no bias in your test (i.e. that there are no extraneous factors outside of the A/B design itself that are affecting the result).  It is widely used in scientific circles.  I have to be honest in saying I've rarely seen it used in the online world - but it is definitely a good approach if you can afford the extra time for the extra test element.

I hope this helps.

Regards