Test Results: Will Reminder Date in Subject Line Increase Article Readership?

When you operate a publishing site like MarketingSherpa, you want to be clear to your subscribers about the content available to them. Articles published in our nine weekly newsletters are “open access” to our hundreds of thousands of readers for seven days before they go into a members-only archive. The barrier date is stated in each email.

Still, results from a survey conducted earlier this year showed that 68% of our readers don't know how long articles are available before going behind the membership wall.

We decided our limited-access policy needed to be clarified. We also wanted to see if adding an expiration date to newsletter subject lines might create a sense of urgency that would boost open and clickthrough rates.

Seems like a no-brainer, right? Never in a million years did we think it would do the opposite. Well, it did. And the lessons we learned are something all marketers can appreciate.

Here are the four steps we followed to conduct the A/B split test and measure results:

-> Step #1. Develop new subject line style

Our standard subject line style starts with the name of the newsletter -- B-to-B, B-to-C, ContentBiz, EmailSherpa and so on -- and includes a few words about the article’s content.

For the test, we added an expiration date in brackets at the end of the subject line for the test groups. This way readers scanning their inboxes could immediately see how long an article was “open access.”

Example:
o Control - B-to-C: Personas Primer
o Test - B-to-C: Personas Primer [Expires 3/26/08]”

-> Step #2. Randomize delivery

Each week, our email production team sent roughly half our newsletter subscribers the articles with the test subject line; the other half got the control. We randomized the delivery options so the tests went to different groups of subscribers each week.

-> Step #3. Provide test ample time

We decided the A/B test needed enough time to generate results that weren’t skewed by the reactions of subscribers to a different subject line style. So, we ran the test for 10 weeks.

-> Step #4. Track key metrics

To determine whether adding an expiration date boosted reader engagement with articles, we tracked five metrics:
o Open rates
o Clickthroughs to the article link
o Total newsletter clicks, including house ads, our jobs posting and all other secondary links
o Email bounce rate
o Unsubscribes

In addition to tracking total figures, we broke out these metrics for the nine different newsletter titles to track potential differences in behavior based on each newsletter’s particular subscriber base.

Well, what did we find out? Turns out adding an expiration date to the subject line of our email newsletters didn’t boost opens or clickthroughs. In fact, it had the opposite effect, contrary to our expectations.

Newsletters with an expiration date in the subject line performed slightly worse than those that used our control subject line in key engagement metrics:
o 0.7% decline in average open rate
o 2.5% decline in average article clicks
o 1.1% decline in average total clicks

At least the difference in the subject lines produced no significant difference in bounce rates or unsubscribes.

Looking at the results by newsletter title, the control subject line beat the test line in almost every metric.

Although we had some anecdotal evidence (a few emails from happy readers) supporting the test subject line style, the message in the metrics was clear: Adding an expiration date wasn’t an improvement over our standard subject line.

In the end, the test was another reminder that marketers can’t make decisions based on gut instincts because you never know how your customers will react. Go with the data.

Useful links related to this article

ExactTarget - MarketingSherpa’s email service provider:
http://email.exacttarget.com/

DeliverySuccess - helps with newsletter production:
http://www.deliverysuccess.com/