Interpreting a multivariate test report

The objective of a multivariate marketing test (MVT) is to discover which factor or combination of factors produces the best result, for example the highest response rate or the most revenue from a marketing campaign. Longbow’s patent-pending expert system for such tests lets you vary up to three factors simultaneously, analyzes the results of the test, and then prepares a report to show you the most effective combination of factors. The objective of this paper is to help you interpret these reports.

Several factors are varied simultaneously in an MVT. Different “recipes” are sent to different customers, with each recipe combining different values of each factor. A two factor test requires four recipes to cover all possible combinations; a three factor test requires eight recipes. Because each recipe includes all factors, you can’t tell which values for which factors are most important just by looking at the responses to the various recipes. This report untangles the effects of each factor and the interaction effects between the factors, showing you which values of the factors will yield the best results in a campaign.

The report consists primarily of two charts. The first chart, a horizontal bar chart, shows you which combination of the factors works best. The second chart, called a Significance graph, shows you the relative influence, positive or negative, of the different factors. For illustration purposes, examples of these charts are shown below.

2x2 Factor plot

For the response rate chart, the two factors being tested are the color of the background (red or blue) and the image (dog or cat). The bars indicate the response rates for the several combinations, and the error range. Combination number two, with red background and a cat image, produced the best response rate, 38.3%. The numbers underneath the factor values, in columns one and two, indicate how each factor contributed to the result. The rightmost column shows the probability that a particular combination of factor values is the best combination. The combination with the highest probability of being the best is shown in green.

The mean response (“overall mean”) was 34.13%. Add to that, for combination number two, the red color effect of 0.63% and the cat image effect of 1.13%, plus the interaction effect of 2.38%.  Adding them up,  34.13 + 0.63 + 1.13 +2.38 = 38.27. In contrast, the dog image in combination number one has a negative effect of -1.13, so that combination did not yield a good response rate.
 
Note that the standard deviation associated with the overall mean (47.4) is much larger than the error bars around the results (4.66) because there is less information associated with the overall mean compared to the information within the recipes of the test. There is variation between the four recipe combinations, and there is variation within an individual recipe. The factors of the test (image, color) supply information on both types of variation, so the amount of variation in the results of the individual recipes is smaller than the variation in the overall mean.
 
All you need to understand the results of the test is contained in this first chart. However you should look carefully to see the relative weights of the various factors, especially when there are several factors. The significance graph (a Pareto-type chart) makes it easier to see the relative weight, or influence, of the factors by displaying each factor and each combination of factors as a bar on a chart, regardless of whether the effect is positive or negative.

2x2 significance plot

As expected from the first chart, the interaction between the two factors has the biggest effect.

Three factor charts

When three factors are tested simultaneously, there are eight recipes. Here is a sample report for such a test. Combination two has the highest probability of being the best choice, at approximately 61%, but combination three at 25% is a reasonable alternative.

3x2 factor chart

Here is the corresponding significance chart.

3x2 significance chart

Of course your report will have different factors, the ones you set up when you created your test. But whatever factors you use, the report will identify the best combination of values for the tested factors, and give you some measure of how much you can rely on the results.