Testing Primer

Why marketers don’t test

Testing is an important and highly visible part of Longbow’s marketing automation service. But many direct marketers avoid testing because until now it has been difficult, time-consuming and expensive.

One difficulty is the need to know some statistics to design tests and interpret results.  Another is that many marketers still believe that only One Factor At a Time (OFAT) can be changed in a test. If a marketer is stuck with an OFAT design, several tests are needed to determine the most effective offers and lists.  Business pressures often don’t allow time for these multiple tests.  If time is available, multiple test runs still require many subjects (customers to try out the offers) and lots of collateral (what the customers are mailed), running up the cost.  For many marketers, it is easier to skip testing and just rely on instinct when designing a new campaign.

Why marketers should test

The main reason to test is to improve the effectiveness and efficiency of marketing campaigns.  Testing reveals the most powerful messages, and the target groups that will be most receptive to them.  These reasons alone should generate sufficient returns on the money and time invested, regardless of the testing methodology used.

The argument for testing becomes even stronger when Longbow is used to manage the process.  Longbow’s expert system approach removes the need to have mathematical and database expertise on staff. Advances in mathematics in the last decade, embodied in Longbow’s patent pending wizards, remove the OFAT requirement so that more than one factor can be tried out in the same test.  This Factorial Design (also known as multivariate testing) lets a marketer test several factors simultaneously and still get accurate answers. Thus the time and cost of testing are greatly reduced

What to test

There are two kinds of tests that give marketers valuable information: targeting tests and offer tests.  Targeting tests determine if campaigns are directed at the right customers.  Offer tests help determine the best message; the one that elicits the maximum positive response.  Targeting tests help the marketer select the most appropriate segment for a campaign.  Possible tests include sending the same communications to different segments, or sending it to one segment and sending nothing to another (control group) segment.

Offer tests compare responses from segments that get different marketing messages.  For example, one segment might receive a 10% discount offer while another receives a 5% discount offer.  One segment may receive an offer displayed against a green background while a comparable segment gets an offer with a red background.  Different offers may have different degrees of personalization.  In all cases, the objective of the test is to see what factors or attributes yield the highest response. The main campaign is then structured to use the best performing values.

When to test

Ideally, all components of a campaign should be tested.  In practice new Longbow users usually roll out their first campaign without testing and later discover that behavioral targeting improves the response rate.  Encouraged by the success of their first effort, the second campaign is typically more ambitious in scope and is often preceded by offer testing.  As users gain experience, they expand their testing suite to include targeting and more offer factors.

Experimental design

Longbow supports two kinds of experimental designs, OFAT and Factorial Design (multivariate testing).  OFAT (also called split run or A/B) has been used for generations.  It’s simple and it works.  Only one factor is tested, with two different values.  It can be as basic as sending one group an offer and not sending anything to another group. The more factors one wants to test, the more time consuming and expensive OFAT tests become.

Because Factorial Design can test multiple factors together, fewer subjects are needed and the testing is completed sooner, reducing the cost.  Effects due to interactions between the factors can also be measured.  For example, a proper Factorial Design could reveal whether using or not using photographs in a brochure works better with bigger or smaller text headlines.

Factorial Design’s power can be offset by the intimidating mathematics used to interpret the results, and by the complexity of setting up the various testing recipes for each subset of experimental subjects.  Longbow removes these obstacles by automatically assigning customers to each group and by doing all the necessary calculations behind the scenes, making two factor and three factor experimental designs easy for marketers.

Conclusion

As marketers bring more scientific rigor to their work in an effort to improve response rates for marketing campaigns, testing becomes an ever more important component.  Longbow delivers the logistical and mathematical support to make sophisticated testing more affordable (time and money) and easier to deploy and interpret.