How to Get 1000% Increase in Opt-ins - Test Results

CHALLENGE
“My conversions for opt-ins stunk,” says Leo Notenboom, Owner, Ask-Leo.com, part of Puget Sound Software LLC. The Ask-Leo site was getting terrific traffic (44,000+ daily visitors) but an “abysmal” subscription rate (only 10-15 newsletter subscriptions a day).

Notenboom also wanted to reduce their reliance on Google for traffic to his website for techies looking for answers. Increasing their newsletter subscribers seemed to be the only clear-cut way of doing it without breaking the bank.

Their audience was fickle by nature, however. Viewers typically came to the site to get a specific how-to nugget of information and then left abruptly. Plus, techies are among the touchiest demographic to market to online.

Many of Notenboom’s peers understand. MarketingSherpa’s Email Marketing Benchmark Guide 2008 says that 11% of emailers state the willingness of people to join new lists is a significant concern. So, how could they get visitors to spend extra time to sign up for their newsletter?

CAMPAIGN
Notenboom started first by investigating alternatives and looking at analytics data. They discovered that the average time visitors spent on their site was 66 seconds.

They decided to test an opt-in process that used a dhtml (dynamic html) time-delay *hover box,* which mimics the actions of a pop-up. Unlike a pop-up, a hover box – sometimes called a ‘slide-in’ – doesn’t get produced by another window being opened. Rather, the hover box code is part of the actual Web page being viewed; it remains hidden for an amount of time to avert pop-up blockers.

“Personally, I had always been turned off by anything resembling a pop-up,” Notenboom says. “But I thought the combination of a [hover box] and a delay was a potential big win.”

A key part of his plan was the delay: not serving viewers with the offer right away but after they took some time to explore the site. But how long should they delay it? They followed three steps to get the answer:

-> Step #1. Set up A/B test

Notenboom and his team knew readers normally stuck around for about a minute before leaving the site. So, they homed in on a time-delay at or near the 60-second mark with A/B testing. They tested three combinations three days apiece -- “long enough for the results to become statistically significant.”

The test combinations:
o 60 vs 75 seconds
o 60 vs 45 seconds
o 60 vs 30 seconds

-> Step #2. Design the hover box

They wanted the time delay to be the only test variable. But they still had to design a pleasing hover box for the test.

Here’s what Notenboom and his team came up with:
o Box size of 35,781 bytes
o Two-tone blue background color
o Two paragraphs of copy in white type

Underneath the copy, viewers were encouraged to sign up by inputting the following information into entry fields:
o Name
o Email address
o “Where you heard about Ask Leo!” (optional)

-> Step #3. Set restriction

They set a restriction on the hover box. Regular readers who didn’t subscribe would see it only every six months at that IP address. The exceptions were users who changed computers or cleared their cookies.

“I didn’t want to annoy my readers. But I wanted to give them a reasonable chance to opt in,” Notenboom says.
RESULTS
The test demonstrated that the time-delay hover box works, and it works best at 60 seconds. It has been capturing a 1000% average daily increase in subscriptions.

Here is specific data from the tests:

Time Delay.....Impressions.....Subscriptions........Rate
60 seconds........196,638.............264................0.13%
75 seconds........188,461.............210................0.11%

60 seconds........157,171.............251................0.16%
45 seconds........149,573.............205................0.13%

60 seconds........201,534.............282................0.14%
30 seconds........185,005.............273................0.14%

“In all of these cases, the difference was small but detectable,” Notenboom says. “For me, what I have learned is that there is a sweet spot for my site at 60 seconds. It surprised me that 60 seconds was better than 45. I expected more people would leave if I hit them later.”

The 30-second delay performed as well as the 60-second one, but the latter was a better fit for his audience. “What I think is happening is that the people who are hanging around and reading more are increasing their perceived value of my site. And then, at the point when the hover box appears, they’re deciding that my newsletter is something worth signing up for.”

Did any of the data surprise him? The pop-up-like annoyance factor that Notenboom was a bit wary of at the beginning has not popped up. “We’ve had no real negative feedback. The hover box has given me the best of both worlds.”

He says the 60-second delay for the hover box shouldn’t be counted on as a magic number, however. “It’s specific to my site, but what the results do show is that the time-delays are definitely worth testing.”

Useful links related to this article

Creative Sample from Ask Leo!:
http://www.marketingsherpa.com/cs/askleo/study.html

Aweber - email service provider that helped with consultation:
http://www.aweber.com/

SplitTester.com - site that they used to calculate numbers:
http://www.splittester.com/

Google Analytics - the analytics system Ask Leo! uses:
http://www.google.com/analytics/

Ask Leo!:
http://ask-leo.com/

Puget Sound Software LLC:
http://pugetsoundsoftware.com/