How to Convert 225% More Trial Subscriptions - Multivariate Test Results

CHALLENGE
Like many premium content marketers, Brad Bacon, Director Distribution & Consumer Applications, Weather Channel Interactive, looks for opportunities to turn visitors to a free website into subscribers for paid services.

Even with strong traffic to the Weather.com site, however, he wasn’t pleased with conversion rates for their Notify! service, which uses email, text and voice alerts to warn subscribers of severe weather in their area.

“It’s a hard product to explain quickly. You almost have to have glanceable content, where you can hit that key benefit point that gets them to register, much less pull out the wallet,” Bacon says.

Bacon and his team had run through several variations for the Notify! landing page. They tried different feature-oriented and benefit-oriented appeals to catch visitors’ interest in a few seconds. But none of those efforts delivered the combination of message and offer that maximized conversions for a free trial.


CAMPAIGN
Bacon and his team turned to multivariate testing to optimize the Notify! landing page. Here’s how they designed and implemented the test:

-> Step #1. Analyze existing customers and conversion path

Before developing the landing page variables to include in the test, the team first stepped back to analyze their current situation. They researched key elements of their Notify! marketing strategy, including:
o Traffic sources to the landing page
o Current conversion rates
o Existing user demographics
o Key selling points and main value proposition
o Customer feedback, including primary reasons for subscribing to the service, favorite features and benefits and key differentiators from other products on the market

-> Step #2. Analyze existing page and develop new template

Next, they analyzed the existing landing page to pull out important elements to test. After that, they developed a new template that employed best practices in landing page design.

Among the elements they deemed important to test:
o Continuity of the Weather Channel brand
o Images to support key messages about product benefits
o Use of customer testimonials
o Offer
o Calls to action

With those elements in hand, they mapped out a page template that formed the basic framework for the multivariate test. They concentrated on landing page best practices, including:
o Compelling headline
o Graphics relevant to the service and its benefits
o Use of a hero shot
o Elimination of navigation options off of landing page
o Short, bulleted text
o Single call-to-action with clickable button

-> Step #3. Develop variations for headline graphics

Research showed that most users subscribed to the service to receive tornado warnings. So, Bacon wanted the page header to feature a mix of text and graphics illustrating a tornado warning scenario to quickly convey how Notify! works.

They tested several different tornado graphics to show how the service could help subscribers get their families to safety. Along with the images, the graphic included three captions explaining the service -- e.g., 1) tornado in your area, 2) Notify! warns you, 3) Your family is prepared.

Options for the graphics included:
o Simple illustrations of a tornado, the Notify! logo and a house
o Photorealistic representations of the tornado and house with lightning added to enhance the bad weather imagery
o Animated Flash version of the illustrations

Testing a Flash version was unorthodox since experts typically warn against using Flash on landing pages. Bacon’s team thought the animated explanation of the service might reduce the amount of text needed on the page.

Still, Bacon was skeptical. “We were very dubious about Flash, but we said, ‘OK, well, we’re willing to test it, but we don’t think it’s going to net out.’ ”

-> Step #4. Develop variations for hero shot

The team had identified space in the left column of the page for a hero shot. They needed to test a range of image options to find one most relevant to the page’s message about features and benefits.

Options for the hero shot included:
o Image of Weather Channel on-air personality who records the severe weather alerts
o Images of different types of severe weather, including a snowstorm and lightning
o Customer testimonials

-> Step #5. Develop options for headline and text

The basic design template emphasized short, bulleted text to quickly describe benefits of the service to visitors who were likely only skimming the page.

They tested different headline options, including:
o “Get Urgent Storm Warnings Anytime, Anywhere”
o “Get Severe Weather Alerts on your Phone”

Beneath that headline, they tested several versions of bulleted text that listed key benefits, including:
o Three to five bullet points
o Bolding certain words in each bullet point, such as “Sleep better” and “Trust the experts”
o Different font sizes

-> Step #6. Develop call-to-action variations

The original landing page offered users a checklist-style comparison between different levels of the Notify! service. Each had a different link to click to take users to the appropriate registration page.

Bacon’s team suspected that the landing page had too many navigation options. Instead, they tested a call-to-action centered on one service level. They made that call to action a single, clickable button and tested several color and style variations:
o Red
o Blue
o Gold
o Three-Dimensional, “jelly-style” buttons
o Flat buttons

-> Step #7. Run multivariate test for one week

After outlining the elements to test, Bacon and his team ran the multivariate test for a week. During that time, they created more than 1,000 different variations of the basic page template to find the elements that worked best together.
RESULTS
The winning combination of page elements delivered a 225% improvement in conversions. Since Bacon and his team were hoping for a 30% increase, they were quite pleased.

Even more surprising than the triple-digit lift in conversions was the fact that the Flash animation version of the header dramatically increased conversion rates over static illustrations. Although they aren’t sure why Flash worked, they think it helped explain a complex product in the few seconds that most users typically spend looking at a landing page.

“One of the real values of multivariate testing is that you’re really customizing the user experience in a way the audience is most likely to respond to,” Bacon says.

Bacon had worried that eliminating the description of different service levels on the landing page might not deliver fully qualified users to the free trial. But analysis of the free trials generated through the new page showed no dropoff in conversion from free-trial to paying customers.

“The point is, we are capturing more consumers on trial and all our other metrics held steady.”

Other interesting results from the test:

- The image of lightning as the hero shot outperformed other variables, such as the image of the on-air personality and customer testimonials.

Bacon’s team did not drop testimonials altogether; they chose to include them in a link on the landing page that opened a pop-up window rather than letting users navigate away from the page.

- The flat, red button outperformed other color and style variations.

Bacon credits the test for convincing his team of the continual value of testing marketing efforts piece by piece.

“We can’t take results and say, ‘Now we need a flat red button on every page.’ For each page, each product and each customer experience, you’ve got to individually test it. Yes, we can’t forget something we learned here, but we can’t assume it’s going to work for every product and every page.”

Useful links related to this article

Creative samples from Weather Channel Interactive's multivariate test:
http://www.marketingsherpa.com/cs/weatherch/study.html

Widemile - helped with Weather Channel Interactive’s multivariate test:
http://www.widemile.com

Weather Channel Interactive:
http://www.weather.com