Are Surveys Misleading? 7 Questions for Better Market Research

by Adam T. Sutton, Senior Reporter

Asking an audience for opinions is a common part of market research. Teams create surveys and form groups to ask consumers a variety of questions.

Philip Graves, CEO, Shift Consultancy, says this approach to market research is a bunk science.

Graves, whose academic background is in statistics, has observed consumers for 20 years; first as a traditional market researcher and later using psychoanalytical techniques. He has advised numerous international businesses, including Pepsi and Whirlpool.

"Over the course of that period, my distrust and skepticism of some of the things I was seeing and some of the things that were accepted as good practice [in market research] grew," he says.

Graves has since published a book, Consumer.ology: The Market Research Myth, the Truth about Consumers, and the Psychology of Shopping, where he emphasizes that opinion polls and consumer questionnaires are as likely to lead marketers astray as they are to improve their results.

We sat down with him to better understand why he says marketers should avoid survey-based research, and what they can do instead. Here are seven questions his opinions raise that you should ask yourself:

Question #1. Can consumers break down their decisions?

Marketers and pollsters regularly ask consumers about their likes, dislikes, needs and worries.

"The assumption tends to be that asking questions of consumers is likely to be in some way revealing or constructive. And I think that simple notion is one that really needs to be challenged," Graves says.

He believes that consumers simply aren't good witnesses to their own behavior and motivations. Even though we're conscious beings, we're capable of being influenced on a sub-conscious level far more often than we realize. That makes it very difficult for consumers to accurately describe their decisions.

Sure, there are times when we agonize over a purchase at the grocery store and can clearly relate our reasoning. But, there are many more times when we grab the same product we've always grabbed, even if we're not sure why.

"We very quickly lose the reason we have the relationship or attachment to that brand," Graves says.

-Factors customers do not realize

Two of the many examples Graves cites in his book are from research by Chicago neurologist Alan Hirsch, founder of the Smell and Taste Treatment and Research Foundation. In the first, Hirsch put one new pair of Nike running shoes into a room with a light floral smell and another identical pair into an unscented room. Of subjects who inspected the sneakers, 84% said they'd be more likely to buy the shoes from the scented room.

In the second study, Hirsch found that wafting a scent into part of a casino led to a 45% increase in money put into slot machines. How many of those people would explain, "I spent more because it smelled nice?" (See links below)

Question #2. Are your questions priming the answers?

Not only does Graves believe people cannot accurately describe what they want, he says they can easily be influenced by marketers' questions, and that surveys can bring misleading results.

For example, when people were asked if they thought the U.S. government should spend money on an anti-missile shield, the results appeared fairly conclusive. Sixty-four percent of those surveyed thought the country should and only six percent were unsure, according to Opinion Makers: An Insider Exposes the Truth Behind the Polls, which Graves cites in his book.

But when pollsters added the option, "...or are you unsure?" the level of uncertainty leaped from six percent to 33 percent. When they asked whether respondents would be upset if the government took the opposite course of action from their selection, 59 percent either didn't have an opinion or didn't mind if the government did something differently. Clearly, questions can change a survey's results.

Graves supplies a hypothetical example: If you ask people whether the U.S. should drill for oil in Alaskan nature reserves, you're likely to get more people to answer "yes" if you first ask a question related to the country's dependence on foreign oil imports than about the environment.

"In the vast majority of cases, that priming is artificial. It doesn't reflect the way people would think of it in the real world. And humans are absolutely [highly] sensitive to priming."

-Prime the experience

The notion that consumers can be primed also applies to the retail and online environment. For example, Graves says, a company that immediately posts a large, red sign advertising a sale inside its stores can help consumers notice sale products by posting similar signs near them.

"That has an effect of sensitizing people who go into the store. They will be far more likely to notice those big, red splashes," he says.

Question #3. Are your focus groups influenced by group behavior?

According to Graves, constructed interviews with small groups of consumers can do little more than misguide a company. People are too greatly influenced by group behavior to accurately describe their opinions or predict whether they'd buy a product in the future.

If you look back at the history of focus groups, Graves says, they grew out of psychotherapeutic groups.

"The whole point of psychotherapeutic groups is that they change people's behavior," he says. "They change the way people think."

In a focus group, the group dynamic is accompanied by "rampant priming," Graves says. Statements by participants and questions from panelists can influence other people's responses. People unconsciously mimic one another to fit in with the group. After they agree, some might become more extreme to seem interesting to others.

"There is a whole bag of psychological dynamics that comes out of group behavior, none of which is conducive to getting to -- accuracy, and all of which is a system that feeds on itself to produce a result that's a byproduct of the fact that you put together the group," he says.

Question #4. Do online surveys reflect purchase decisions?

Online surveys are a quick and cheap way to learn about your audience -- so the thinking goes. But Graves' opinion is that the surveys are too far removed from the environment and mindset in which consumers purchase.

"They don't reflect the frame of mind people are in with they're consuming. They reflect the frame of mind of someone in front of a computer screen."

For example, an online survey about advertising should not be presented to consumers shopping for books, Graves says. People do not go to online book stores to evaluate their purchase decisions. They go to explore books. The survey would reach this group in the wrong frame of mind, and according to Graves, people are capable of acting in completely different ways when that happens.

"It's like the souvenir you bring back from a holiday. When you're on holiday and you're in that environment, it feels like the most exciting thing in the world. When you get it home, it sits on the shelf and gathers dust, and pretty soon you start to wonder what the heck you bought it for," he says.

Question # 5. Can you replace "asking" with "observing?"

Monitoring people's behavior, particularly when they don't realize they're being monitored, offers the best market research, Graves says. This includes website analytics, purchase data and any similar stats.

"Genuine behavioral data is always true. It's always what people have done," Graves says.

In online retail, for example, website data is gathered in the same environment in which consumers make purchases. Sales data, too, is gathered in the proper context. The data is not opinion -- it’s fact.

If you want to understand consumers when they're shopping -- watch them when they are shopping. They will be in the correct mindset. Do not ask them about their shopping habits while they're reading the news online.

- Tests are the best

Instead of using online surveys, online marketers should run A/B and multivariate tests, Graves says. Brick-and-mortar stores should watch how people shop in their stores.

"You're testing in an environment that replicates all the variables that are in the experience," he says.

Question #6. Are you testing all the variables?

Another problem Graves sees in much market research, particularly research done before a product launch, is that it often does not focus on enough of the factors that affect consumers' purchasing decisions.

Business-to-business companies, for example, may demo products to potential customers and ask if they'd use them on a daily basis. Beverage companies might hold taste tests to see if a product is palatable and ask people if they'd buy it.

The problem, Graves says, is that the above examples only focus on the product. They do not account for the product’s pricing, packaging, availability, competitive context -- and a whole host of other factors that affect purchasing decisions.

Even a product's label and region of origin can affect consumers' perceptions. In his book, Grave cites two Cornell University studies where consumers were given the same wine in different bottles. In one study, those drinking from the bottle with a Californian label rated the wine's taste higher than those drinking from a bottle with a North Dakotan label. The second study confirmed the findings (see links below).

"It's ironic that businesses invest all sorts of money into sales and marketing -- and they do that because it can influence what someone actually buys. But to then strip those away when deciding what to launch -- along with packaging, branding, price, etc. -- doesn't make sense."

Question #7. How confident are you in the data?

Statistical analysis often comes with confidence indexes to help researchers understand the data's reliability. But before you try to understand data's statistical reliability, Graves says you should understand its "psychological reliability."

To accurately convey reality, consumer data must come from consumers in their natural purchasing mindsets and environment. Graves uses the acronym "AFECT" to help gauge whether a set of data is from the correct psychological context:

o Analysis -- is the data an analysis of behavior? If they are self-reported opinions and attitudes, you're in problem areas, Graves says.

o Frame of mind -- was the audience's frame of mind similar to those people you're trying to influence? For example, if you're an online apparel retailer, is the audience comprised of people shopping for clothes online?

o Environment -- was the audience walking on a street corner or using a search engine? Is this where you will find your target audience?

o Covert -- did people know you were monitoring them? If so, it likely influenced their behavior and fouled your results.

o Time frame -- did the time in which the audience members made a decision accurately reflect the time in which customers will make a decision?

Useful links related to this article

SherpaBlog: Marketing Research and Surveys: There are no secrets to online marketing success in this blog post

Members Library: Six Scientifically Proven Ways to Succeed in Office Politics

Members Library: Pricing Psychology Test: Shopping Guide Lifts Order Value 35%

Focus Groups vs. Reality: Would you buy a product that doesn’t exist with pretend money you don’t have?

New York Times: Scent and Sensibility (5th paragraph)

Opinion Markers: An Insider Exposes the Truth Behind the Polls

Science Daily: Wine Labels can Ruin a Meal

Consumer.ology: The Market Research Myth

Shift Consultancy