June 06, 2013

Event Wrap-up

Optimization Summit 2013 Wrap-up: Top 5 takeaways for testing websites, pay-per-click ads and email

SUMMARY: Overlooking the misty Boston Harbor in an unusually chilly May, marketers met in a modern day salon, cozy with like-marketing minds, eager to jump into discussions their co-workers shy away from -- with 23 sessions covering topics such as A/B testing, value proposition and lead generation.

Read this event wrap-up of the top five takeaways from the days filled with tests, insights and plenty of clam chowder at MarketingSherpa's third annual Optimization Summit.

by Courtney Eckerle, Reporter

Every corner you turn in Boston is packed with historical significance. You can hardly walk a few feet without running into plaques and statues dedicated to the men and women who made contributions in shaping the course and culture of America.

Inspired by the city, the challenge of the week quickly became how marketers could change the history of their own companies — or, in the case of our keynote speakers, actually changing American history — with testing. Then, taking what they’ve learned back to their offices to become a positive influence in the daily tasks of their co-workers, as well as the overall culture.

We have compiled five robust takeaways from the 23 sessions at MarketingSherpa and MarketingExperiments Optimization Summit 2013 in Boston to help you improve your results and perhaps even transform your marketing department’s culture.

Takeaway #1. Think of value proposition as the center of everything you do

With the largest newsroom in the nation, The Boston Globe faced the herculean task of turning its staff into optimizers at a critical moment in print media’s history.

"It's really no surprise to anybody, our industry is completely disrupted ... we're at this point where we really need to re-engineer our business model," said Peter Doucette, Executive Director of Circulation, Sales and Marketing, The Boston Globe.

The Boston Globe has ensured their legacy as one of the oldest and most respected newspapers in the country for 141 years — this was a great foundation and history to build off of, Doucette said, "but what we've done historically no longer cuts it … we don't want to end up as a black dot [which represented failed newspapers in Doucette’s presentation], because what we do is too important."

The paper has a history of being a digital pioneer by launching Boston.com with free content in the '90s. To complement its free digital property, the company launched BostonGlobe.com a year and a half ago to offer premium paid content. Doucette said it wanted to continue to re-invent its brand and monetize customers.

"We were trying to take a different approach to the same fundamental challenge that is facing print media," he said. The mindset then became, "the flip side of every challenge is opportunity."

Going forward, The Boston Globe will continue making a new evolution in their long history to "think of value proposition as the center of everything we do," he said.

The starting point

With optimization, there is always an opportunity to make your site even more effective at driving subscriptions and incremental revenue.

"It's really about a starting point, and starting that evolution, and the way we figure out how to scale this is through testing and optimization," Doucette said.

Testing is a process that includes all staff, he added, "not just a start and then it's over." The team’s radical business model redesign included four steps:

Establish your "why"

Determine your market

Distill your value proposition

Prep to test your value proposition

The team created this new digital subscription business, and created a distinct value proposition for their determined target audience, and began to, "really build testing plans, and optimization plans that are relevant for their respective flows."

The team focused on eight areas vital to the customer experience to test:

Asset converting the most subscriptions

The checkout process

Price

Incentive pricing

Value proposition

Email/Push communication

Offers through social channels

Nurture

Doucette said the team let the customer experience inform how they began, and currently approach testing, and understanding the customer life cycle became the foundation of all testing.

Takeaway #2. Develop a testing culture

A main facet, Doucette said, is to create test plans that adequately communicate what you are trying to accomplish, and when.

Setting up collaboration and experimentation to shift the company culture is key, he added. Begin by knocking down walls with those who deal closely with customers in order to integrate customer data points by tapping into phone calls or looking at online behavioral data. Testing has brought in almost $3.6 million, and the driving force behind that has been a culture change, he said.

Editorial headlines, especially, were the perfect place to test. "It's an ecosystem that's built for testing," Doucette said. "Ultimately, we're squinting at the horizon to shorten our decision windows, and become a real decision leader."

The team began A/B testing article headlines live, empowering their own editors and writers to be a part of the process, not just the analytics marketers.

Through testing, it became evident headlines that were less formal or more personalized usually performed better than those that strictly adhered to journalistic rules.

For example, "Oh yeah: Time for the Sox" received a 117% higher clickthrough rate than "Red Sox gear up for spring" and "Dismissal of Lynn teen’s murder conviction upheld" had a 60% lift over the headline that didn’t list the subject’s town.

The team has tested more than 351 headlines since September of last year, and Doucette said, "thinking about testing through the entire business creates opportunity, but we also have to create results from that."

Keep data front and center

"Data was always front and center, every decision was going to be data driven. But we still had to do a lot to feature that culture of testing, and get it beyond the data geeks," said Amelia Showalter, Director of Digital Analytics, Obama for America, in the keynote speech Tuesday evening.

"Testing a lot of times wasn't about one big, much better version versus a much worse version — two percent here and four percent there, it starts to really add up," she explained.

Showalter listed four strategies for creating a testing culture that will lead to an overall successful testing program:

Use every opportunity to test. No email went out without at least three test treatments.

Compete against yourself. Don't focus on what the competition or rest of the industry may be doing. Ask, "Are you getting better than you were last month? ... We were comparing against ourselves … I don't think it even occurred to us to try and find out what [Gov. Mitt] Romney campaign's click rates were."

Keep a testing calendar. "If we saw a blank space, it prompted us to have a brainstorm session to have more tests ... to keep that culture of testing constant."

Circulate results internally. The campaign staffers kept an internal Listserv to increase buy-in and familiarity. Circulating those results “increased the literacy of everyone on the campaign, and it sometimes prompted discussions,” she concluded.

By focusing on these four aspects, staffers, even those not necessarily involved with testing, would come with different theories on findings, or make a case to re-do or continue a test.

Develop a SWAT team of testers

The question Ashish Braganza, Senior Manager, Global Business Intelligence, Lenovo, posed was: How do you get your organization to have a financially viable testing practice?

"Getting that executive support is pretty critical," he concluded. "You are your own best cheerleader for your practice — you play the role of evangelizer."

There are so many business units, he added, that you need to talk it up and create buzz around your optimization practice to get buy-in from co-workers as well as executives who hold the purse strings.

The best way to become a cheerleader is to assemble a team you want to brag about and rally others behind.

"Testing and optimization is not the cavalry ... it's the SWAT team," he said, listing his SPADE method to assemble the right group to cover all areas.

Strategy: singular focus that aligns to goals

Project management

Analytics

Design and content

Engineering

He listed project management as especially critical, headed by a diplomatic team member with the people skills to take charge and gain consensus for launching a test.

"Find the one person on your team ... who can come to the table with solutions," he said.

This team needs to be able not only to conduct tests, but read the right takeaways from them — not just throw data at a problem.

"Launch a test and then it's analysis paralysis ... have fewer KPI's and keep it simple, trying to find that upfront before you go into the test, and not afterwards," he said.

Takeaway #3. Listen to your audience — get out of the marketer's mindset

This could also be called the "I was wrong" takeaway. Nearly every session leader remarked testing is a humbling pursuit. A marketer has to prepare to be totally and utterly baffled by the results, and only one rule is concrete when it comes to testing — your audience has the final vote.

"Don't trust your gut" was definitely one lesson learned from the 2012 campaign for Toby Fallsgraff, Email Director, Obama for America.

An example of this was when marketers in the Obama office began betting on the outcome of tests — a fun game called the "Email Derby" to involve the office in testing soon turned out to be a lesson in humility for the staff.

"It turned out we were terrible. We were worse than [random] chance," Showalter said.

She added that "even the experts can't reliably predict what will win and what will lose. It was a really good lesson ... let your audience tell you what you want to hear."

Fallsgraff said he was one of the worst in the office, and he "quit somewhere in the middle because it was too embarrassing."

He said an intern was actually doing the best at predicting the test outcomes, but "to be fair, she wasn't doing any better than a monkey pressing buttons, but she was much better than I was."

Every email is an opportunity to learn more

Every email the Obama campaign sent might have three or four treatments, Showalter said, and sending upwards of 20 emails a week nationally adds up.

"If you use each one of those as an opportunity to test something, you can learn a lot quickly," she said. Tests they would run in emails might be:

Subject line and draft tests

Full-list tests

Background personalization tests

Consistently testing was important because "even the things that were paying off were [eventually] losing their luster," Fallsgraff said.

For instance, strategies that had been successful in the 2008 campaign no longer were, and successful tests they ran in the beginning of 2012 often lost their novelty.

Fallsgraff and Showalter attributed $2.2 million in additional revenue from sending the best treatment of an email compared to the worst.

Even if they sent the average result, Fallsgraff said, "We'd be leaving $1.5 million on the table."

Pretty doesn’t always pay

"We couldn't trust our guts. We decided and found out pretty early on that being sleek and pretty didn't pay off for us," Fallsgraff said.

He estimated they had around 20 people whose job it was to "make things look prettier and sleeker, and that worked in a lot of areas, but it didn't always work in email."

When sleek and pretty wasn’t doing as well for them, they asked the question: what about ugly?

Much to their surprise (and chagrin), the ugly combination of tests and links worked, and the control with "a whole bunch of legal text" also won, Fallsgraff said.

"What pained us mostly was we tested yellow highlighting," he said, which kept winning over the prettier tests, until it eventually lost and he joked, "we discovered there was a God."

Challenge assumptions — The Boston Globe test:

Doucette described a particular test where they challenged assumptions by taking a strategy that had worked for a similar venture — the digital subscription process for The New York Times.

The Boston Globe tested out an accordion checkout process that had proved successful for The New York Times. Despite the similarities that would have indicated this change would be successful, the test came out with a 35% decrease.

"It just didn't pan out," Doucette said, and the team realized that their long form checkout process was better for their audience.

His takeaway was to "capitalize on your failed test to get a learning and build a win for the next test … It's about the cumulative impact of what you're doing."

The Boston Globe's overall results to-date speak to their testing with every aspect of the customer’s journey, with $3.6 million going into their second year with paid content.

AARP tests content to adapt with its audience

AARP has 37 million members over 22 million households, and "that is a huge audience to learn from," said Gaurav Bhatia, Vice President, Digital Strategy, AARP Services.

"Listen to members, and test what they are telling you," he said about their testing strategy. The 50 years or older group is tech savvy — they are the fastest-growing audience on Facebook, and 53% of Americans aged 65 or older use the Internet or email, according to a Pew Internet & American Life Project Survey.

AARP needed to deliver value online to its audience, and used tests to determine what would provide the best usability.

The six tests the team presented all focused on making digesting content easy — easy to find, share, understand, listen, access and use.

Responses from a site survey showed that their audience found the site content challenging to read, and showed the very desire the Pew Research indicated — members were interested in sharing via social media.

In an effort to reduce site abandonment and exit rates, and drive awareness with social media, AARP created a treatment that increased overall font size of text on the site, changed the background color, and made articles easier to share by incorporating social media icons.

By adapting to its audience's desired experience, AARP was able to see a 12% decrease in page bounce rates, a 7% increase in email sharing and a 3% adoption of social sharing.

Members also felt there was no easy access from the homepage to the products and content they really wanted, and to lower the high usage of their 800 number, AARP decided to test a “Quick Links” treatment.

Adding quick links for products made finding information on the site intuitive and easy for members, resulting in a 25% increase in Web referrals as members quickly found the path to their product, as well as decreasing calls into the (1-800) number.

CNET cuts half of their newsletters, and increases engagement

Diana Primeau, Director of Member Services, CNET, did not want to cut newsletters at the beginning. She was diligently following the instructions of a superior by taking a step back and looking at CNET’s 26 newsletters for redundancy.

With solid engagement metrics for the overall portfolio, the assumption for years had been that CNET subscribers were having a positive experience.

To start off, they asked two questions:

Does what we’re doing make sense?

Is it still relevant to my customer?

The answers to those questions surprised Primeau and her team. They discovered several newsletters that were either no longer relevant to their current content, or newsletters that contained duplicate content with another too-similar newsletter.

Not only was what they were currently doing not working for their subscribers, but the editorial team was doing a lot of extra work that "wasn't paying off."

Quality, not quantity

Primeau said CNET needed to focus on providing value to their subscribers. For too long they had been distracted by engagement metrics, but needed to "take a step back" and view the customer experience overall.

Theteam had become focused on the marketing aspects of newsletters, and needed to evaluate each newsletter through understanding the user experience each provided, from internal content to how each mapped back to the site.

So, in order to optimize what they were offering, Primeau and her team eliminated newsletters that:

Were left over from an old acquisition

Shared content with other newsletters

No longer had current content to draw from

Had low engagement rates

Primeau and her team lost less than 5% of their subscribers in the transition, and increased clickthrough rates in their existing emails by 13%.

CNET was not only competing with "all the other mail someone gets in their inbox," but it had been competing against its own sends as well.

Aside from freeing up the editorial team's time to create content of higher value, by getting out of their marketing mindset Primeau and her team are providing subscribers with a better experience from CNET newsletters.

Takeaway #4. Don’t be afraid to get personal in content

"I don't think big data is Big Brother. Using data and optimization helped us listen to our audience and [we] learned what they like and how they wanted to participate," Showalter said.

The interactions with the Obama for America audience that were "the most personal and the most human, all of that happened to win," she added.

The fear of most marketers in utilizing personalization never appeared for them. She said, "People didn't find it creepy."

They addressed subscribers by their first name, kept an informal tone in emails and used "drop-in sentences" that referenced people's past behavior, for instance prompting a donation based on an amount already given — "Donate $61 or whatever you can before tonight's fundraising deadline."

An example of the informal tone that worked best for them appears first in subject lines, with some of their most successful being simply, "Hey" or "Hell yeah, I like Obamacare."

"Content matters — as an email director, this is one of my favorites," Fallsgraff said.

The content of their emails was written in a direct and personal tone, either from the president himself, the first lady, or staff members of the campaign that each had a voice of their own. This provided a real connection between subscribers and the frontlines of the campaign.

"It felt like there were real people behind those machines. There was an honesty and an authenticity behind those communications," Fallsgraff said, adding that people were "actually upset or getting emotional" when Finance Director Rufus Gifford sent out his final email communication.

Time-of-day testing

Showalter mentioned that they did test time of day to see if a specific time worked particularly well for their audience, but it didn't yield very productive results.

"I think it's one of the least productive testing options you can do. Maybe not for some ... but we tested it a few times and basically all it said was don't send really, really late at night, and don't send really, really early in the morning," she said.

Takeaway #5. Test value proposition in content to transform decision-making

Embracing the aha moment of testing was an important step for Jon Ciampi, VP, Marketing, Business Development & Corporate Development, CRC Health Group, and his team.

"We were putting all of this money into it, and I'm getting beat up every day," he said, adding that best practices weren't enough — they needed to see the big picture from the customer’s point of view.

In a multi-factor split test for an addiction and mental health rehabilitation facility, Ciampi and his team were able to realize what was most important to their customers. They had been working under the assumption that luxury would draw their audience in, and developed content that way.

However, it was trust that emerged as the previously under-utilized element that was trumping luxury with their customers.

"The whole marketing team went, 'Woah! They want to know if this is safe, if they can trust us,'" he said.

Coming to this conclusion involved testing their "Free Assessment" page. The control was a short-form page template, with an above the fold call-to-action, and the treatment was two times the length of the control, included much more information and placed the call-to-action at the bottom of the page.

By utilizing a single-column, long-copy approach, the treatment was better able to guide the prospect’s thought process, and ended up generating a 22% increase in total conversions.

CRC Health was able to incorporate this new knowledge into other tests — including a multi-factor split test on a working cattle ranch for troubled teens.

Opposed to the glamorized imagery and "unique setting" focus in their control, the treatment featured trust aspects, such as:

Trust-based messaging: "recommended by doctors and therapists," "24 years in business"

Testimonials featuring real people

A picture of the director with a direct message

Third-party credibility indicators

By embracing a learning (or aha moment) from a previous test, they were able to apply it to multiple tests in different areas and further confirm the validity of the original findings — and see a 29.8% increase in total conversion.

Value proposition in PPC ad A/B split test

With the success of trust-based content in previous tests, Ciampi said his team decided to apply it to PPC advertising for an addiction and mental health rehabilitation facility.

With an objective to increase the clickthrough rate, they designed an A/B split test that pitted "company logic" versus "customer logic" in their value proposition.

Company logic was "we have the most doctors, therefore we have the best care," and the customer logic was fear-based, and was a person looking to know who they were trusting their loved one to.

1. Controls

Branded PPC ad, "Sierra Tucson Care Center: Considered a Top Recovery Clinic. Get a Free Assessment. Call Now!"

Non-branded ad, "AZ Alcohol Detox Facility: Exclusive, Luxury Rehab Facility. 1 Clinical Staff Per 3 Patients."

2. Treatments

Branded ad, "Sierra Tucson Care Center: Considered a Top Depression Clinic, Traditional & Alternative Therapies."

Non-branded ad, "Arizona Drug Rehab: Considered a top Addiction Clinic, Traditional & Alternative Therapies."

The branded PPC ads received a staggering 14,000% increase in clickthrough rate, and the non-branded ads saw a 3,300% increase in clickthrough by focusing on value proposition.

Understand the psychology of words

The word "clinic" versus "center" has a strong connotation, Ciampi said, and using "center" creates a difference of over three times the clickthrough rate. The first comes across as quick, cheap and informal, while the other conveys expertise, research and educated.

Understand through testing how the keywords or phrases you are using and including in content to convey value are going to resonate with your audience.

Word choice in content reflects the mindset the team realized consumers needed, Ciampi said, and further included "nurturing," "understanding" and "we" words in their content.

CRC Health’s audience is "highly empathetic, social, caring, worried and thus wanting a more formal, highly educated service with high interaction."

Social media can have a big impact on your value

The team at CRC Health's success in one area placed pressure on another area of the funnel — social media.

Poor reviews on social sites such as Yelp became a sore point in an otherwise successful online value presentation to customers.

In order to turn a one star rating into a 3.5-star rating, they were forced to double their reputation management and resources into:

Rating monitoring

Proactive updates

Suppression management

Team members undertook the "very difficult task" of reached out to clients to ask for reviews, or interviewing them and based off of that, sending them a review to edit and post as they wished.

"It doesn't take a whole lot [of more ratings] to get your ratings up, but it does take a lot of work," Ciampi said.

From one test, CRC Health was able to implement the learnings across the board, focusing on communicating trust and compassion as the main value provided to customers throughout the entire funnel — from its main page and even into the call center, he concluded.