How Interactions in your Analytic Experiments cause low confidence results

September 17, 2009 by Gregg Larson

How many times have you set up a test only to find that no matter how long the test is run, you can never develop a high enough confidence with a result to pick a winner? When you analyze the data however, you can see that there are clear winners at intermediate steps? A good example of this is web you are running a test that requires someone to click on a "Add to Cart" button before they buy the product. You may be running an A/B Split test or a multivariable test and notice that certain combinations are clear winners in getting people to click on the "add to Cart" Button, however, when measured against the conversion rate of sales, no winner in determined, and you are stuck with low conversion rates. No matter how long you run this test your outcome wil remain inconclusive because of an interaction unknown to you.

The good news is that you actually have two events that are significant winners, the bad news is that you have them in a tug of war. Your results are only going to be as good as the experiment you design. Unfortunately, there are more ways to do it wrong than there are to do it right, and without the experience and statistical background to detect these interactions, you can easily design into experiments interactions that will hide all the good work you've actually done. How do you avoid this? First is by understanding the underlying assumptions regarding the Chi-Squared Test for Independent Proportions -or just ask a statistician. The most important assumption when running these tests is independence. As I will discuss, there are many ways you can create dependence between processes, and there are ways to create independence. These are the tasks of the experimental designer.

What is an Interaction?

An interaction is best shown graphically:

As you can see, the lines cross each other. It is at this point that the variables share a relationship and can no longer be analyzed independent of each other. As my professor always told me "Statistical software is not robust against youreself!": web analytic tools like Omniture or Google's Web Site Optimizer won't be able to tell this, and won't understand that you've designed a test that breaks the underlying assumptions it is expecting!

Permalink