About Multivariate Testing

What it is

Multivariate testing (MVT), sometimes called multivariable testing, is a technique for testing hypotheses on complex multivariable systems, varying several variables simultaneously to determine which combinations yield the best results, which variables have the largest effect on the results, and which interaction effects (if any) exist among the variables. MVT can be used to simultaneously detect the effects of numerous variables on the outcome of a marketing campaign.

History

With the rapid rise of interest in multivariate analysis, many people are surprised that its origins go back to the 1870s. Principal component analysis dates back to 1901. Factorial designs are over a hundred years old. However, the mathematics are computationally intensive, so the methodology had few practitioners until modern computers became widely available in the 1970s to handle the calculations. Usage broadened in 2008 when Google released its web site optimization tool and Loyalty Builders introduced Longbow, which has an expert system for MVT, enabling marketers without training in statistics to implement MVT designs and interpret the results.

What MVT reveals

• Which variables are the most influential
• Which variables (if any) actually have synergy or conflict between them (interaction effects)
• Which combination(s) of variables are most effective
• How significant the difference is between the combinations
• How the combinations perform compared to business as usual

Why it’s not used more often

Until recently, MVT has been used only by sophisticated mathematicians. The reasons for this neglect are three-fold:

1. Bad education -- marketers (and many scientists) were taught for years that if you changed more than one variable at a time, the results of your test would be inconclusive.
2. Complicated mathematics -- even those who knew the first reason was wrong had to struggle through difficult, math-intensive processes.
3. Lack of intuitive graphics -- once the testing was complete, the interpretation of the results, especially interaction effects between the many variables, was just as difficult.

Why use is increasing

Despite these difficulties, MVT usage is increasing because of some compelling applications. First is email. With its low cost per email sent, it is feasible to mail in volume and thus collect credible statistics. An email campaign is over in 36 to 48 hours, so it doesn’t take long to get numerous runs completed. Also, it is quick, easy, and inexpensive to change important factors such as subject line and offer, so email is close to a perfect environment for MVT.
A second appropriate application for MVT is web site design, especially for sites with moderate to heavy traffic. Again, it is easy to change the variables and the statistics are  abundant. Further, measuring clicks is an excellent way to test the variations. Google has taken advantage of this situation by offering a free, effective, and hence very popular MVT tool, its website optimizer. This increased visibility for MVT has undoubtedly led to its use in other applications.
 

MVT and factorial design

A test is actually an experiment, and for MVT there are two commonly used experimental designs: Full Factorial Design and Fractional Factorial Design. Full Factorial Design uses every combination of factors to run an analysis. Fractional Factorial Design uses a subset of all the combinations of factors to extract useful information in an economical fashion. There are pros and cons to each method.
 
The Full Factorial Design delivers complete information, but when more than three factors are being tested simultaneously, the number of runs needed for a valid test becomes cumbersome. The Fractional Factorial Design delivers incomplete information, but if the combinations are chosen strategically, the lost information may be minimal or even trivial. With two or three factors, Full Factorial Design is recommended, and with four or more factors, the recommendation would be to fractionate.
 

Comparing MVT and A-B split testing

To illustrate the differences between the methods, we have prepared a detailed example of how they would be applied in a real life marketing campaign.

The Loyalty Builders approach to MVT

To overcome the many obstacles to MVT, Loyalty Builders created an expert system for MVT inside Longbow so that marketers who were less mathematically sophisticated could use this powerful methodology. The Longbow tagline (“We do the math. You do the marketing.”) is especially applicable to the testing module, for it hides the mathematical complexity of MVT while enabling advanced test design. Testing is an integral part of mathematical marketing.

The expert system can handle up to three factors (variables) with a full factorial design, which is a comfortable number for most purposes. For those applications that need more simultaneous factors, the Loyalty Builders team of mathematicians and analysts will build fractional factorial designs on a consulting basis.