Testing and Optimization: Variance testing for B2C contributes to 45% lift in conversion rate

A lot of discussion goes into the importance of testing, but where A/B testing gains a lot of attention, variance testing is often put on the back burner.

However, that isn’t the case when you talk to Emily Emmer, former Senior Interactive Marketing Manager, Extra Space Storage. Emmer spoke about her experience with variance testing at MarketingSherpa MarketingExperiments Web Optimization Summit 2014.

Emmer describes her first few years testing with Extra Space Storage as the Wild West of testing. Because there were no set rules for the company’s testing, campaigns and proposals would often involve chaos.

"You know, when we might set up a test or consider a test or want to turn a test off, there were as many different opinions as people in the room," she said.

Extra Space Storage is a publically traded company that was founded in 1977. The brand serves a nationwide audience with more than 1,000 self-storage properties and generates over $1 billion in revenue annually.

According to Emmer, Extra Space Storage encourages an aggressive testing culture and has several mediums to test on, including the brand’s site, mobile app and call center.

"Our goal is about a 30% increase in conversion rate annually, which, as tester, you know is not an easy feat," Emmer said.

Extra Space Storage’s lack of definite rules made for a difficult testing environment. Emmer turned to variance testing to remedy this problem and create some guidelines for testing. Variance testing measures two identical experiences to determine the natural variability of the site.

Emmer and her team ran five simultaneous variance tests in an attempt to answer:

• How long do we run our tests?


• How much traffic do we need for each treatment?


• How many treatments do we run in our tests?


• How big must the lift be to call it a winner?


• How many conversions do we need per treatment?

"I was able to take this information and then extrapolate that into rules that my organization could use moving forward," she said.

After developing and implementing these testing rules, Extra Space Storage achieved a 45% increase in conversion rate from the previous year.

Watch this session from Web Optimization Summit 2014 to learn how you can start to utilize variance testing in your culture, when you should start considering variance testing and the five tests Emmer recommends running.

Related Resources

Reserve your seat for MarketingSherpa Summit 2016 — February 22-24, 2016 at the Bellagio Resort in Las Vegas

Marketing Mashup: Top takeaways from Web Optimization Summit 2014

B2C Marketing: Site and search optimization leads to 15% increase in average order value

Testing Optimization: Voices.com realizes a 33% improvement in sign-ups

Email Marketing Chart: Mobile versus desktop conversions