Common reasons for low confidence results in your website tests

October 10, 2009 by Gregg Larson

You are testing too many things at one time: Whether you are testing more than two pages in an A/B test, or running a multi-variable test, the more variables you test, the more samples you are going to take to get a high confidence result.  The problem is that by design, MOST of the items you are testing are not the prime driver!  The more you test, the larger the haystack you are going to have to look for.

The Fix: Instead of  running one test that ecompasses everything, find a way to reduce the number of variables per test and run a series of tests.  As I have said, when you are done with a test, review not only the results of the test, but also the external factors, and base each sequential test upon these new inputs.   When looking at the test results, while you may not have a high confidence winner, you could have a high confidence LOSER!  Use my test validator to understand if this is a possibility.

Permalink