by
Selena Blue, Manager of Editorial Content
How do already effective marketers get even better?
That's the question MarketingSherpa wanted to answer with the Email Summit 2015 Live Test.
This year, we interviewed three national nonprofits to find a new client for our test. A key quality we were looking for was a marketing program that was already doing well.
VolunteerMatch was selected as' a unique nonprofit organization that receives most of its funding by selling a SaaS product to a B2B audience. Before Summit, we featured
a case study of the nonprofit's previous success with decreasing its sales cycle.
Tessa Srebro, Lead Generation and Marketing Associate, and Lauren Wagner, Senior Manager, Engagement, both of VolunteerMatch, worked with Jon Powell, Senior Manager, Executive Research and Development, MECLABS Institute, to plan and execute the live test.
THE CUSTOMER
VolunteerMatch actually serves three different customers:
- A B2C audience of people who want to volunteer
- Nonprofit organizations looking for volunteers
- A B2B audience of businesses looking for employee volunteer program solutions
For the live test, the team wanted to focus on this last audience.
Wagner explained that by offering a Software-as-a-Service product through VolunteerMatch Solutions, the organization is able to self-sustain without fundraising — a unique ability for a nonprofit.
"Companies we typically work with already have an established employee volunteer program. Perhaps they outgrew the current system they were using to manage this program," Srebo said.
She added that the ideal companies have more than 5,000 employees "because we've found smaller companies don't have a designated budget set aside for their employee volunteer program or just a small budget."
The team also looks at people, not just the companies they work for. Looking at job titles, they typically work with people from the following departments:
- Marketing
- Human resources
- Community outreach
- Corporate social responsibility
"Those are the people we're really looking to connect with," said Srebro."
CHALLENGE
The struggle for the team is that the highest value opportunity exists within a smaller email list. Less than 200 companies out of a list of 9,000 shoulder the responsibility of funding the entire volunteer-to-organization connection through the VolunteerMatch product solutions.
That creates one driving question for the team: What is the most effective way to attract and connect with the right companies who fit our customer profile?
CAMPAIGN
As part of the Summit live test, the MECLABS and VolunteerMatch teams had a few objectives. First, they needed to answer the driving question about attracting ideal companies despite the small list size. Second, the team had to design a test that would engage the Summit audience and provide valuable takeaways for them to apply to their own tests.
The team set out to do that in a three-part testing series where each test connected to each other in the form of a design of experiments. They would then execute each test and analyze them to determine customer insights to apply to later tests.
Step #1. Analyze data of current efforts to uncover testing opportunities
First, to understand its customers and prospects, as well as the marketplace, the team did a comprehensive analysis of VolunteerMatch and its competitors. By learning how current prospects and customers engage with the VolunteerMatch, the team could spot areas where testing could reveal opportunities to not only learn more about customers, but also better serve those customers.
For example, if visitors are clicking all over the place, perhaps the current funnel isn't as streamlined as it could be.
The team examined the following:
- Visitor website clicks and paths
- Traffic sources to site
- Organic and paid search keywords
- Competitor product offerings and messaging
- Competitor search results
The team then cross evaluated those opportunities against science-based messaging methodologies to discover additional opportunities. Through the thorough analysis, 35 testing opportunities were uncovered and prioritized. The top ones were then used to create a series of experiments.
Step #2. Create a series of experiments that build upon each other
Because the email list for the SaaS side of VolunteerMatch was limited, the team had to think outside the box for how to gain some insights before hitting that list up at Email Summit.
In the first test, they wanted to try and discover what value emphasis would draw the most people into a conversation about the corporate solutions product.
The team "decided that a B2C cross-sale email test focusing value proposition might be the most effective in helping us achieve our goals," said Powell.
In a second test, they wanted to understand the most effective approach to inspiring action, or clicks, after someone responds to the email and learns about the product. They would achieve this through a landing page test where the B2B companies visit on the Solutions website.
In a third test to be performed live at Summit, the team would take the
insights gained from Tests 1 and 2 and allow the audience to help construct treatments for an email test to the B2B email list.
Click here to see the full version of this creative sample
Step #3. Use subject line test to learn about potential value points
Because the B2B list for VolunteerMatch Solutions was on the smaller side, the team knew it would be a challenge to run a successful, statistically significant test. To overcome that, the team decided on a less orthodox approach. They would use the much larger B2C list of causes and volunteers.
The team developed a subject line that would help discover the most relevant aspect of the VolunteerMatch Solutions product, at least in the eyes of the B2C list. This could help point them in the right direction of what messaging the B2B list might would respond to as well.
In an effort to bring in fresh ideas,
the team turned to the audience of two blogs, Moz and MarketingSherpa, for suggestions for three different value points for the product.
The team provided the audiences with three categories of value. Each would have nearly the same email but with the bolded portion of the last sentence customized to each value. Based off the submissions, they also included a fourth category: passion.
Click here to see the full version of this creative sample
Click here to see the full version of this creative sample
Two subject lines were selected for the three categories, as well as one subject line for the new passion category. The passion subject line was run twice, to act as a double control to flag potential validity threats.
Category #1: Proof
- Attention Business Leaders: How to Increase your ROI through Employee Volunteer Initiatives
- Volunteering matters. We have the proof.
Category #2: Network size
- CC Your Boss: 1,000+ Ways To Make A Difference (Inside)
- Does your company care? Thousands of ways to prove it.
Category #3: Ease of use (app)
- The volunteer app your coworkers will talk about
- The One App That Can Change The Way Your Company Gives Back
Category #4: Passion (no feature)
- Spread the Only "Good" Office Virus
- Spread the Only "Good" Office Virus
Test 1 Results
"One of the things that we were really surprised about was that the network didn't surge ahead as the big winner because, internally, one of the things we always focus on as our big differentiator in the market is that we come with this network 99,000+ nonprofits and all of their volunteer opportunities to deliver to your employees right off the bat when they first sign into the site," said Wagner.
The Network Size category together saw a 12.5% decrease in click rate compared to the best performing category, passion. The other two categories did not validate a statistical difference against passion.
However, when we compare each category's subject lines against one another, it appears one significantly outperformed the other. From this, the team was able to see one pattern emerging:
"Subject lines with volunteer or volunteering in the beginning performed a lot better than those that didn't. This makes sense to us. These people are volunteers. Volunteering matters to them. It's something personal to them, part of their lives. If they see something volunteer or volunteering in the subject lines, then it resonates," Wagner explained.
Click here to see the full version of this creative sample
While the control pair of passion subject lines performed best as a category, as individual subject lines, they didn't come out as the winner. However, the team could still gain insight from the wildcard category.
"If we're talking about a product, we need to offer them a tangible benefit or product offering. … that was a great learning for us," Wagner said.
Step #4. Execute landing page test to determine most effective CTA
Instead of looking at product-level value, which was hit upon in the subject line test, Test 2 sought to better understand motivation and behavior behind those B2B visitors with even the slightest interest to learn more. This would help the team to better understand which action(s) should be asked of customers in the third experiment, the email test.
With that in mind, the team analyzed ways to improve the call-to-action on the Solutions landing page. The team made three changes.
Click here to see the full version of this creative sample
First, "Contact Sales" became "Speak to an Expert." This change was an effort to lessen anxiety around clicking the button. Visitors may not want to be "sold to" by Sales, but they would be open to talking to an expert.
Second, visitors would be clicking for "custom pricing," rather than just a quote.
According to Srebro, "It sets them up for what's on the next page because the word 'custom' shows that you're going to need to tell a little bit about yourself. … We think they'll be more likely to fill out the form if they know what they're getting into."
Third, a second CTA was added to provide customers not ready to commit to speaking to someone yet. While having dual CTAs can sometimes split traffic in a way that hurts overall conversion, the team thought offering a lower-commitment would entice more clicks overall.
Srebro explained why this was still a good thing for VolunteerMatch, saying, "If someone clicks on the demo, we still have the opportunity to collect their contact information so we can continue to engage with them, keep them educated and interested … until they are ready to speak with someone."
Test 2 Results
The team's expectation that the treatment would win was confirmed with an increase of 291% in overall clickthrough.
Click here to see the full version of this creative sample
One interesting find was that 30% of those who clicked on the "Speak with an Expert" filled out the form, while no visitor who went to the control and clicked on "Contact Sales" converted.
Step #5. Apply previous discoveries to email test live at Summit
On Day One of Summit, Powell enlisted the help of the audience to apply the insights gained from the previous tests to a final email test. Powell, Wagner and Srebro gave an overview of Tests 1 and 2 to the audience before giving them their assignment.
Reflecting on the results of Test 1, Wagner said, "What will be interesting to look at is with our corporate solutions list, we are looking at people who are typically employee engagement professionals, so these are people who are managing employee volunteer programs within their company. So what we're wondering is if that some finding will be mirrored within our corporate list, with the term 'employee engagement' because that's something that is personal on a day-to-day basis with them as well."
In groups, the audience was to pick one of the three predetermined categories used in the subject line test: Proof, Network of Opps and Ease of Use. They then had to craft a subject line, headline, optional sub-headline and opening paragraph of an email to the solutions email list.
The team tweaked the suggested subject lines so that they fit a similar structure, with the value point of each category being the sole difference. Below the opening paragraph, the remainder of the email would stay the same for each category treatment. The two CTAs from the winning treatment in Test 2 were used as hyperlink CTAs in the body of the email.
Click here to see the full version of this creative sample
Click here to see the full version of this creative sample
Click here to see the full version of this creative sample
Test 3 Results
In terms of open rate, there was no statistically significant difference between the subject lines given in the sample size.
However, clickthrough did provide some insight. Despite being unable to calculate relative difference between all three categories, the proof category appears to have achieved the greatest response when compared to both Network and Ease of Use categories. Because of the clickthrough for Ease of Use, we can't compute the relative difference.
A relative difference between Proof and Network, however, showed a 75% increase for Proof, making it the winner of Test 3.
Click here to see the full version of this creative sample
The results are already helping the team rethink how they do some things.
Wagner said, "The network didn't win, which made me a little sad. However, that's really great for us to know because we really focus on it so much, but it's nice to know maybe we should be focusing on other things."
She went on to say, "We've had a hunch for a while because we hear from our clients all that — from these people who are managing these programs — that it is so difficult for them to get the internal buy-in they need, to get the funding they need for their programs, to prove to their managers, their VPs and their CEOs that employee volunteering really makes a difference and it's a really important program to have."
Srebro explained this is because the team wants to connect with specific people, not just companies. The team has gained valuable information about their target audience: The people they're dealing with are hungry for anything that will help them prove the value of employee volunteer programs.
This has already led the team to brainstorm content ideas and ways to get case studies from current customers to help prospective clients to make the case for utilizing VolunteerMatch.
Creative Samples
- Design of Experiments
- Category #1 and Category #2
- Category #3 and Category #4
- Test 1 results
- CTA Control and Treatment
- Test 2 Results
- Treatment 1
- Treatment 2
- Treatment 3
- Test 3 Results
Sources
VolunteerMatch MECLABSRelated Resources
Email Marketing: How a B2B company lifted conversion 200% with personalized email messagesWebsite Optimization: Testing program leads to 638% increase in new accountsB2B Lead Gen: A/B split test helps increase quote requests 262%
