Marketing Research Chart: 75% of strategic marketers use A/B testing to learn about customer behavior

by Daniel Burstein, Director of Editorial Content

"To stop trying to think ahead of the search engine algorithm changes and focus more on changes in consumer behavior."

The above quote is from the MarketingSherpa Website Optimization Benchmark Report survey, in which a marketer shared her top lesson learned about site optimization.

In that same Benchmark Report survey, we asked marketers:

Q: Does your organization use website optimization and/or testing to draw conclusions about your customer base?

Click here to see a printable version of this chart

Experienced marketers leverage split testing for customer insights

Overall, 47% of marketers said they use website optimization and testing to draw conclusions about customers.

But when we segment the data, it gets interesting. The more mature the marketing organization, the more it is able to use A/B testing to learn about customer behavior.

As you can see in the chart above, 76% of trial phase marketers — those who do not have a process or guidelines for optimization or testing — do not use testing to learn about customers and build a customer theory.

However, 75% of strategic phase marketers — those who do have a formal process with thorough guidelines routinely performed — use testing to learn about customer behavior.

True website optimization is difficult …

The reasons for this are clear. Website optimization and A/B testing require your department to:

• Create (at least) 2 of everything you want to test


• Have the right technology in place to ensure the traffic is split correctly


• Design experiments with a thorough understanding of the scientific process to ensure you're really learning


• Understand statistics (or, at the very least, know about the proper statistical processes) to calculate validity

It can be much easier to just ask customers through surveys, focus groups or customer service and Sales interactions, for example.

But who is willing to admit:

• "I didn't buy your product because you're a bad salesperson.”


• "I chose your competitor because your advertising claims are laughable and I don't trust your company."


• "I'm far too lazy to learn how to use this new technology. I only said I would buy it in the focus group because I wanted to sound like I knew what I was doing."

Sometimes, people don't tell you because they don't know themselves.

Who even knows to say:

• "If I bought that car, I would be afraid that my peers' perceptions of me would change."


• "I'm overwhelmed by the available options, so I will choose inertia over action and not buy anything."


• "Anytime I see anything that mentions the stock market, I'm scared of losing money so I decide to keep my money in the bank instead of investing."

… but enlightening

As one marketer replied in the Benchmark Report survey:

"Ask you customers what they want (easy). Design experiments to test their assumptions and perceptions about what they want (harder, especially if these also test your own assumptions. Are you even aware of your own assumptions?) Believe what your customers tell you based on their behavior in the experiments."

Your own — personal, company, department, organization, agency, etc. — assumptions often need the most testing, as this marketer replied when asked what lessons she learned:

"That all unknowns can be found with testing, and more often than not, the answer is very counterintuitive. Our worst enemy is in the mirror."

Use all customer intelligence to your advantage

This isn't to say you shouldn't conduct surveys. Or ask for your customers' opinions. You should.

But take it with a grain of salt.

Use these opinions to help test real-world actions. Opinion-based information gathering can be used to:

• Inform experiment design


• Interpret experiment results

After all, just because you have identified a behavior, it doesn't mean you understand why customers are acting that way. By combining what they tell you with what they actually do, and putting your own assumptions to the test, you will begin to understand the most puzzling marketing challenge of all — human behavior.

Related Resources

Web Optimization Summit 2014 in New York City — May 21-23

Customer Theory: What do you blame when prospects do not buy?

Customer Theory: How we learned from a previous test to drive a 40% increase in CTR

Testing: Go big, or go home?