Marketing Research Chart: Does A/B testing deliver a positive ROI?

by Daniel Burstein, Director of Editorial Content

While it would be valuable to really know the most effective marketing messaging, let's face it, A/B testing takes a lot of work. You need:

• The IT resources and tools to create and measure a split test


• To create two versions (at least) of every element you want to test


• A testing methodology to ensure you run valid tests, and learn from those tests

Is all of this extra work really worth it? To find out, in the MarketingSherpa 2012 Website Optimization Benchmark Report, we asked marketers …

Q: Did optimization or testing demonstrate ROI in 2011?

Click here to see a printable version of this chart

Many marketers are not measuring the ROI of testing

The majority of marketers could not or did not calculate ROI for conversion rate optimization and A/B testing. Calculating that ROI can be difficult, and it is an imperfect science. You have to include not only, for example, a software platform purchase, but also the extra time a designer had to take to produce not only the original, but a treatment version as well.

Calculating the ROI can be extremely valuable. After all, demonstrating return on investment is the way marketers are able to secure the resources to continue to run the tactics that are effective for their companies (and, dare I say, the way marketers are able to earn promotions after proving their success to business leaders).

If you don't know, well now you know

If you are among the marketers who aren't measuring split testing ROI, here's the good news — it is very likely positive.

After all, 38% of marketers demonstrated a positive ROI with their testing efforts, for 6% of marketers testing was a wash, and only 3% experienced a negative ROI.

And, that ROI can be extremely powerful …

$200 million in extra revenue

When I asked Amelia Showalter, Director of Digital Analytics, Obama for America, about the amount of campaign donations that could be directly attributed to improvements thanks to A/B testing, here's what she had to say.

"When we're working on the campaign, we're actually working so hard to run all those tests that we didn't always keep perfect track of exactly what results were long term. It's hard to calculate this stuff out when we want to put all our resources into running more tests. So, we don't actually ever have a perfect estimate of actually how much extra revenue was due to our testing, but I think that $200 million is a fairly reasonable estimate."

So, if you aren't calculating the revenue of your A/B testing you can take heart that, yes, it is quite hard, even for the advanced professionals.

You can also take heart that if you're running valid tests, you are likely improving the bottom line.

So, give it a try. While your first calculations won't be perfect, and you don't want to take too many resources from running the tests themselves, those ROI numbers can help you gain the resources you need to continue, and improve, your testing.

Related Resources

Email Optimization: A discussion about how A/B testing generated $500 million in donations

MECLABS Online Testing Course

Website Optimization: Simple A/B test adds 'thousands and thousands of dollars' to Cars.com's bottom line