How MVT differs from A/B split testing

To illustrate how MVT differs from A-B split testing (the common methodology), we compare how you would use each method to test several aspects of your marketing campaign against the business as usual (BAU) campaigns that have been historically run. The three variables (“factors”) that we want to test are 1) image used; 2) accompanying text: and 3) headline. Suppose you have limited the choices down to two new versions in each variable, image 1 vs. image 2, generic company message text vs. product specific text, and a headline 1 vs. a headline 2. Below are the steps involved using MVT and A-B split testing, and the relative time associated with each.

A-B split testing: Segment your mailing population in half. Send one group the collateral with Image 1 and the BAU text message and headline. Send the other group the collateral with Image 2 and the BAU text message and headline. Wait 30 days (or however long your campaign runs) and record the results.

Suppose Image 1 was significantly more effective in terms of response rate. Now, repeat the process but this time use Image 1, use generic company message text on one group and product specific text on the other group, keeping the BAU headline. Wait 30 more days and record results.

Repeat the process with differing headlines. Wait another 30 days and record the results.

Now (at least 90 days later if you use direct mail) you know which image is most effective, which text message is most effective, and which headline is most effective. Or do you? Are you certain that results would be similar if you used image 2 with the two message schemes? These are definite possibilities. You could do additional testing, but a quarter of the year has already gone by and some of your customers are being over mailed. An alternative is MVT.

Multivariate testing: Segment your customers into eight groups, namely:

Group	Image	Text	Headline
1	Image 1	Text 1	Headline 1
2	Image 1	Text 1	Headline 2
3	Image 1	Text 2	Headline 1
4	Image 1	Text 2	Headline 2
5	Image 2	Text 1	Headline 1
6	Image 2	Text 1	Headline 2
7	Image 2	Text 2	Headline 1
8	Image 2	Text 2	Headline 2

Mail to each combination, wait 30 days, and find the combination with the best results. You can also use advanced mathematics (or Longbow!) to uncover any interaction effects between the variables. Finally, see how the better performing combinations compare to BAU.