Best Day to Convert Trial Signups: Test Results

CHALLENGE
Getting Web viewers to sign up for 14 days of free brain-training games and exercises has been the top of the sales funnel for Lumosity since its launch in July 2007. Joshua Duyan, VP Marketing, and his team have been charged with enticing trial users to come back and pay $80 a year for an annual subscription.

Lumosity’s business model is built on consumers getting hooked on the proprietary “brain games,” which are designed to bolster memory, mental sharpness ad basic cognitive abilities.

Subscription sales got off to a nice start last year, but Duyan wasn’t satisfied with response to their email-driven campaign. “We felt like we must have been emailing people at times where they actually could not play our games. That was a dilemma we needed to solve.”

Duyan and his team faced an issue that all emailers have been attempting to answer -- the “when” of contacting consumers. “We wondered if we would do a lot better if we emailed people on the day of the week in which they signed up as opposed to the Tuesday-through-Thursday approach that industry data often advocates.”

CAMPAIGN
Duyan decided that testing was the only concrete way to determine when they should email past brain-game players. At the time, the site had been live for three months and they were truly learning on the run.

“How we measure effectiveness is through our ‘usage’ statistics, which represents how many people are actually playing the games and is chiefly driven by email,” Duyan says. “That statistic is more meaningful to us than clickthroughs, and we wanted to see it improve.”

Here are the five steps they followed to test their email program:

-> Step #1. A/B split test

First, Duyan decided to conduct a simple A/B split test to determine whether the sign-up day mattered.

- Control: This group received emails according to industry standards -- their original strategy. These recipients received messages on either Tuesdays or Thursdays, depending on when a campaign was ready.

- Test: This group received the same exact emails -- but on the day they originally signed up for their free trial. If a player signed up on a Sunday, for instance, they received their email on the same day of the week for the duration of the test.

-> Step #2. Tie test to back end

Next, Duyan and his team created a test group based on the setup of their back-end system. When readers are added to the Lumosity database, they are tagged with a signup date and put into separate day-of-the-week folders.

After the promotions were readied, emails were automatically sent to each day-of-the-week folder. (e.g., people in the “Sunday folder” were emailed on Sunday.) This allowed them to do the test without taxing the marketing staff.

-> Step #3. Use extended test time

They ran the tests far longer than normal (six weeks) so they could be sure of the results.

->Step #4: Send at same time of day

The team also sent emails at the same time of day -- noon Pacific Time -- based on the results of an earlier test. That time had proven to be the winner in an earlier test, which pitted several send times against each other: 1 a.m., 10 a.m., noon, 4:30 p.m. and 9 p.m.

-> Step #5. Choose a bigger sample

Duyan took 4,000 names from their list of non-responders and split it in half to conduct the test -- a much bigger test group than usual. Non-responders were past brain-games users who had gone at least a week without visiting the website.

RESULTS
Duyan’s test results prove that conventional tactics should be challenged. Consumers who received emails on the same day of the week that they originally signed up “won handily.” Also, game usage at their website jumped 81% -- it’s the metric that drives sales and is deemed most relevant to increasing paid subs.

Since implementing the results into their program, subscriptions have been “growing consistently,” Duyan says. “What this proved to us is that mailing them on their signup day makes sense. I think this is especially true if people on your list live a very scheduled life, where they are able to spend time at the same point in the week -- almost every week.”

Other lessons learned:

- They’re getting more feedback
Reaching out to readers on their original signup days has spawned a significant increase in unsolicited feedback on what features their subscribers would like to see added to the site. “For a startup like us, that is extremely valuable information to take into consideration,” Duyan says. “And we have gotten great testimonials that we are using on our website, too.”

- Always test new concepts
Testing is paramount to evolving a company’s email program. “If we hadn't tested this idea, we’d probably be emailing on the best proven day for all our customers as a whole rather than emailing them on the day of the week they showed a likelihood of having time to spend online.”

- Test as large a group as feasible
“After this experience, I’d be very hesitant to make a decision based on only 500 or so addresses because we found the numbers can really fluctuate. If you have the list to test 5,000 people, it is worth it.”

Useful links related to this article

Creative samples from Lumosity:
http://www.marketingsherpa.com/cs/lumosity/study.html

MailChimp - Lumosity’s email service provider:
http://www.mailchimp.com

Lumosity, part of Lumos Labs Inc.:
http://www.lumosity.com