Taguchi Methods and Web Analytics

January 19, 2010 by Gregg Larson

Within the web analytics world, I have noticed two common themes regarding the  "Taguchi Methods" and how they are used within the Web Analytics Space.  I was working with a client whom had mentioned they had wanted to use the "taguchi methods" in order to test the optimum pay per click ads for their keyword programs.  The first thing he had confusion about is the difference between multivariate testing, A/B testing, and Taguchi functions.  The second thing he asked me about was why some of the tests created through Taguchi Methods look so bad.

Having studied Genichi Taguchi and Edward Deming during my Master's Work in Applied Statistics, as well as performing a Capstone project which made use of Orthogonal Arrays, I hope to offer some advice as to what "Taguchi Methods" really are, their most common usage, and what this means to multivariate testing in web analytics.  With this understanding, we'll dive into why some of your test cases look so bad, which will introduce us to some common pitfalls that can occur when using these methods without a clear understanding of what they do.

What is a Taguchi Method?

Taguchi Methods is simply a trademarked term that describe a set of statistical experiments used to improve quality and reduce costs.  They include a very specific form of multivariate testing where the combinations are pre-determined based on a set of factors that you wish to study.  One of the most important features of Taguchi methods is that, when designed correctly, will allow you to perform statistical analysis that is rich with data, while in standard multi-variate testing you may be limited in the amout of statistically valid data.  Taguchi methods also allow you to study a subset of the entire problem, while still providing statisically valid conclusions.

While his work in orthogonal arrays greatly eased the development of orthogonal arrays at a time when there were no computers, the reality is that computers can now create these tests in a matter of seconds, making his contribution less important than it used to be.  That being said, Taguchi's main contribution to industrial design and experimental design was his method known as the "Taguchi Loss Function" which calculates the cost to your company for not developing to the nominal value of your specification, even if you are developing product "to spec".

How does this apply to internet multivariate testing?

Many people I interact with tell me "I am good at A/B and multivariate tests in Google, but my clients ask me for Taguchi Methods".  I tell them they've been using Taguchi Methods the whole time!  An A/B test is just the simplest form.  There is a reason to understand what these tests do, and how they do it, and that is so you can write tests that produce meaningful results.  I firmly believe most cases of tests which produce no statistically significant results is due to the fact that the tests were designed incorrectly.

Why do some of the tests look so bad?

Taking a step back, it is important to understand what a Taguchi Method Experiment is attempting to accomplish: What is the most critical attributes in your test?  In order to figure this out, a set of experimental combinations are put together that may or may not represent actual offers or settings.  These combinations and the resultant measurement, when analyzed using statistical procedures, will  give you this answer.  Therefore, it is fair to say that you may see a test that represents all the "best" attributes as equally as you may see a test that represents "bad" attributes.  Both are critical to a successful test.  Remember, these tests are designed for industrial experiments, where a company agrees to undergo a test, and, as a result, throw away the samples created from the tests, even if they turn out better than the original product. 

The first mistake made In the Internet Marketing world is these tests are more than likely run on a set of live customers, which means that if you give a "bad" test to a customer with a propensity to consume, you may have just lost revenue to your site - and that is a risk many internet marketers refuse to take.  Unfortunately, this also invalidates the entire test - because a key tenet of all statistics, including the Taguchi Methods is that the samples analyzed were obtained randomly.  Random means that you understand your sample population, and all have the opportunity to be chosen to take your test.  Random is not people clicking on an add.

Taguchi done right

Having studied Taguchi while obtaining my Masters in Applied Statistics and Engineering Management, and used the Taguchi Methods for my capstone project, I am keenly aware of the pitfalls of these designs, and am trained to extract the most amount of relevant data from a properly designed experiment.  Contact me to understand how to design better experiments and learn how you can improve the results of your tests.

Permalink