November 30, 2006
Case Study
SUMMARY:
Traditional A/B split testing has long been marketers’ secret weapon. Testing a single element at a time makes it possible to be certain which one works best.
But what if you need to test several elements to pin down what works? Or maybe you suspect that making two or more tweaks at the same time could produce synergies that yield even greater gains than one change alone?
Here’s a real-life case study demonstrating how harnessing multivariate testing let one troubled site rapidly sort through many possibilities and apply the results to improve conversions. Includes headline creative, samples and results.
|
|
CHALLENGE
“We were actually thinking about selling our site to some other marketer who might be able to do a better job at it,” confides Sumantra Roy, an owner in ParrotSecrets.com.
More than 83% of all visitors were bailing within 30 seconds of entering the site, and sales were far below what Roy expected. Roy believed in the product -- a set of four ebooks on training and caring for parrots -- and traffic was substantial, but much of that was generated by expensive keyword buys. Even sign-ups for the site’s free enewsletter were a meager 0.4%.
After two years of struggling, it was time to stop squawking and take action. Roy and his team were determined to reduce the abandon rate and improve conversions. But was the problem the copy? The offer? The site’s look or element positioning? All of the above?
After analyzing the site and looking at competitive sites, they uncovered seven factors and 20 variations on each that they thought should be tested. With traditional A/B testing, this would mean 1,152 sequential tests that would take 24 years to complete. Clearly, a more practical approach was needed.
CAMPAIGN
The team employed multivariate testing that would require just 19 tests in two phases, over two months.
Phase I:
In this phase, they tested seven factors and variations of the site’s landing page:
1. Headline: The marketing team brainstormed 15 benefit-oriented headlines based on their own expertise and their analysis of competitive sites and came up with three variations to be tested against the control:
- Control Headline #1: “Inspire Your Parrot To Happily Talk, Perform Amazing Tricks, Glow In Health, Stop Biting Forever, Stop Screaming Forever ... In Short, Become The Dearest, Perhaps Even The Funniest Pet You Ever Dreamed To Own!”
- Headline #2: “Bring Your Parrot's Bad Behavior To A Grinding Halt In Only 3 Simple Steps ... And Transform Your 'Feathered Rebel' Into A Talking, Tricking, Healthy, And Hilarious Stand-Up Comedian!”
- Headline #3: “If You Love Your Parrot Enough To Dedicate 15 Minutes A Day, I'll Show You A Treasure Chest Full Of Proven Parrot Training Secrets That Work ‘Bird Wonders’ Of Pure Pet Fun And Joy ... Watch Your Parrot Talk, Trick, Play ... And STOP All Bad Parrot Behavior ... Right Now!”
- Headline #4: “Easily STOP And Prevent All Parrot Problems ... Like Biting, Screaming, Or Any Grief In Less Than 15 Days ... And Easily Train Your Parrot To Talk, Trick, And Play Like A Joyful Child That Loves To Have Fun With You.”
2. Subhead: The existing page had no subhead, so they decided to test adding a customer testimonial subhead.
3. Price: The control price was $59.95. Based on their competitive analysis, three additional price-points were also tested: $39.95, $79.95 and $99.95.
4. Discount percentage: Given the high margins in selling digital books, the site was offering a 50% discount to encourage immediate response. Two alternate discounts, 40% and 60%, were also tested.
5. Discount method/time limit: The original pop-up simply said the discount “will end by midnight.” They tested two alternates: an “introductory offer” that “will end in less than 1 hour” and a “marketing test” that “will end in less than 1 hour.”
6. Position of newsletter sign-up form: The newsletter sign-up form was positioned near the top of the landing page, just under the headline. This was tested against a newsletter with a sign-up form near the bottom of the page, just before the copy pointed out the product’s cost.
7. Background color: The site’s original background color, blue, was tested against a red background.
All of these variations were fed into the testing software, which sorted through the thousands of possible variations and created 16 test versions. (Roy says it took a month to do the HTML setup work on all the options that were to be tested.)
Each of these was a different combination of the factors and variations. The first test contained Headline #1, a testimonial subhead, $39.95 price, 40% discount, “introductory offer” discount wording, newsletter subscription form positioned near the bottom of the page and blue background.
The second version contained Headline #2, a testimonial, $79.95 price, 50% discount, “good till midnight” discount wording, newsletter subscription box positioned near the top of the page and red background.
The 16 test versions —- and the control -- were rotated across the site, so that each was viewed by an equal number of visitors. About four weeks of testing produced enough data on the number of book sales and newsletter sign-ups from each variation.
This data was fed into the software program, which created a short list of three variations statistically most likely to yield the best results. Interestingly, the program determined that none of the 16 versions (or the control version) was optimal and spit out three new, best-bet versions for further testing.
Phase II:
This second, one-month phase involved classic A/B split tests of four variations -- the three new best bets, plus the site’s control version:
Control: Headline #1 (see above for full wording of headlines); no subhead; $59.95 price; 50% discount; “ends at midnight” wording; newsletter sign-up form near top of page; blue background.
Test Version 1: Headline #4; $79.95 price; subhead present; 50% discount; “introductory/expires in 1 hour” wording; newsletter form near bottom of page; red background.
Test Version 2: Headline #4; $79.95 price; subhead present; 60% discount; “introductory/expires in 1 hour” wording; newsletter form near top of page; red background.
Test Version 3: Headline #4; $79.95 price; no subhead; 50% discount; “marketing test/expires in 1 hour” wording; newsletter form near bottom; red background.
RESULTS
“The final, winning variation increased revenue by 85.1% -- more than double what we’d hoped for in our wildest dreams,” Roy exclaims.
Phase II’s clear winner was Test Version 1. Using this variation resulted in the revenue increase in just a few months. Actual orders increased 40%. At the same time, newsletter sign-ups jumped by more than 600%, from 0.4% to over 3%.
Not surprisingly, visitors abandoning the site within the first 30 seconds also dropped significantly, from 83% to 55%.
“Our results revealed several points, starting with the fact that we’d actually been charging too little,” Roy observes. “The perceived value of the product actually seemed to increase when the price was raised to $79.95, and that meant not only better sales, but also a significantly better profit on each sale.”
Roy could attribute 28% of the revenue increase to the higher price point and 72% to the headline, background color and other changes.
The success of the red background “is counter to conventional marketing wisdom, which indicates that red is supposed to have potentially negative psychological implications. This just demonstrates that you never know what will work for your particular product or site until you test it.”
Useful links related to this article
Creative samples from both phases of the ParrotSecrets.com tests:
http://www.marketingsherpa.com/cs/parrotsecrets/study.html
The Conversion Multiplier - the multivariate testing program used for the tests (Sumantra Roy is CEO of Conversion Multipler):
http://www.ConversionMultiplier.com
Dick Shaver - the database marketing/customer relationship consultant who partnered in creating The Conversion Multiplier:
http://www.dickshaver.com
ParrotSecrets.com:
http://www.ParrotSecrets.com
