Test and Learn: Prove Relevance

In many respects, history is a lot less relevant than it used to be to today’s marketers. A lot that’s “new,” with little historical context, is shaping the way brands are built, businesses succeed and marketers must operate. There’s new media, of course. And new sorts of customers, as well—best described as smart and vociferous skeptics. Too, there are new approaches that combine the traditional with new strategies and tactics to create optimal outcomes.

Combined with senior management’s mandates for performance that satisfies business objectives, it all spells a vastly more complex management challenge than marketers have ever faced before.

In this sort of environment, why, then, are so many marketers using traditional, historical or one-dimensional modeling techniques to measure the outcomes of their initiatives? In an environment characterized by almost non-stop change, metrics built around returns on past investments and based on lagging, versus leading, indicators may not be accurate gauges of future results.

Marketers may be better served by what can be called the “test and learn” approach to initiatives and evaluating outcomes. It involves systematically varying all elements of the marketing equation using experimental designs to determine the optimal marketing strategies for a brand, product or geography.

The test-and-learn approach involves experimenting with different combinations of tactics—new and traditional, together and separately—on a smaller scale and with varying parameters, such as price and product mix, to evaluate results. The various combinations trialed are based on both judgment and fact-based assumptions about which blend of tactics will create the greatest financial impact or increased efficiencies.

The resulting metrics can be extremely powerful in building the case for brand investment. They can drive faster in-market learning to support corporate strategy hypotheses. And when tolerance for marketing missteps is low and budgets are static, test-and-learn can also offset much of the risk that keeps marketers from stepping outside their traditional comfort zones.

It’s all about marketing effectiveness. And make no mistake: This is an issue that’s holding marketing back from wielding greater influence in the organization. Prophet’s “2006/2007 State of Marketing Study” on marketing effectiveness found that no marketing activity was measured consistently consistently by more than 54% of participants. More than one-third admitted: “I do not understand how to best use and leverage new media to meet my business objectives.” Most telling, 84% admitted that marketing return on investment is not well understood within their businesses. Test-and-learn may not be the panacea for all that ails too many marketing organizations, but it will go a long way toward bringing about a cure.

Minnetonka, Minn.-based UnitedHealthcare, a Prophet client, is one concern that has used test-and-learn modeling, partly to find the most effective means by which to grow its business, but also to demonstrate marketing’s ability to drive results.

“We changed price, product and place across the board and then accelerated promotional activities in some markets,” recalls Chris Gibson, UnitedHealthcare’s chief marketing officer. “That helped us prove how much changing price moved the mark, compared to how much additional business growth we achieved by creating the right marketing and advertising mix.”

Gibson notes that different combinations of print, radio and outdoor advertising were tested and tied to business goals. The resulting data proved out which tactics were most effective. But it also “helped many people move up the learning curve to understand that marketing is not just communications—rather, it can be a driver of growth,” she says.

The effort also helped UnitedHealthcare better shape its marketing investment mix, Gibson adds. “Once we had supporting data on how to most effectively communicate our brand and benefits across a variety of tactics, we took that combination and deployed it in 20 additional markets to test its success. And it worked,” she says.

In today’s environment, marketers need to learn how to make optimal use of all the tools available and the tools opening up to them, and in ways that leave them less vulnerable to risk of failure. Applied on an ongoing basis, the test-andlearn approach to marketing investment is ideal for creating the quick wins that build marketing’s credibility while also providing a platform for continuous optimization of the overall mix. Ultimately, that will help expand the purview and influence—and prove out the relevance—of marketing throughout the organization.