This morning, my rep texted me from the shop where my car is serviced:
“You’ll receive a survey from us. It would mean a lot to me if you completed it. We hope you recommend us.” I get he wants a positive review, but this is an example of clear survey bias.

Essentially, my rep is pushing for the answers he wants his managers to see. This is not scientific data collection. It’s public relations or marketing. Either way, it’s what we call Gaming and it’s the second flaw on the survey bias list below.

I frequently negotiate fees with my rep, so I have everything to gain by giving him a good score. Since all I care about is getting the lowest fees, it’s simply not in my best interest to provide the most honest feedback. Basically, even if my rep is rude, I’ll give him a top score to stay on his good side for the best pricing.

Is the point of the survey for customers to get good pricing?

Or, is it for the company to measure the quality of their customer service?

Why Issue Surveys?

Why ask your customers to take surveys if the data you collect doesn’t capture real facts?

Bad Customer Surveys are Ubiquitous

Earlier this year I wrote about how Whole Food’s point of purchase receipt suffers from a complicated, disorganized survey request. I also explored how Alaska Airlines is pestering its customers with surveys that ask more than 94 questions.

Moreover, in 2016, we conducted a study of the surveys issued by the nations’ largest retailers. With one exception (7-Eleven), the surveys by the top retailers were unscientific and failed the basics of customer engagement.

Basically, bad customer surveys are not only ubiquitous, they’re a multi-billion-dollar industry!

Let’s Imagine a World with Good Surveys

Let’s imagine a world in which companies ask meaningful questions in meaningful ways.

In this world, companies would ONLY use surveys to interact authentically with their customers. They would put a kibosh on leading questions. And the would never bug customers with irrelevant or rote inquiries.

With this goal in mind, I have compiled a checklist so you can vet your own surveys for survey bias. There are 4 main survey biases, within which there are 20 types of flaws.

 Survey Bias: Bad Data from the Start 

Have you ever had a contractor bring a washing machine to your house and show you a card that says, “Please give me 5 stars my job depends on it!”? Or maybe you bought a big-ticket item and were told they could come down in price for a good survey score? Garbage in. Garbage out. If you think your company may be collecting flawed data, use mystery shoppers to document which customers are asked to take your survey — and how.

Flaw #1: Mis-Representative Samples

This is when your survey data only comes from certain kinds of customers and doesn’t represent your customers at large. For example, your data could come from those with lots of free time, those with a specific gripe, or those who gave their email addresses. Basically, with misrepresentative samples, your survey sample omits one or more customer groups—or your survey sample is too small to give you reliable facts you can count on.

Flaw #2: Gaming

This is when associates only survey customers they believe had positive experiences. They also might ask customers to answer the survey in exchange for implied or explicit favors.

 Survey Bias: Poor Question Wording 

How questions are phrased has everything to do with whether you will get accurate data and whether your customers will engage with your survey. The best way to clean up wording flaws is to work with an objective team, ideally outside your company. But if that’s not possible, at least work with another department.

Flaw #3: Leading Questions & Statements

These questions prompt customers for the answers you want to hear. Both tone (when done over the phone) and content are used to push customers toward particular answers. For example, “How satisfied were you?” assumes the customer was somewhat satisfied.

Flaw #4: Double-Barreled Questions

These questions ask two things at once, so it’s unclear which question the customer is answering. For example, “Was your rep efficient and proactive?” asks about two different behaviors in one question.

Flaw #5: Forced Questions

These questions ask your customers to say something about themselves that may not be true. For example, “What’s your favorite gym?” when the customer may not use a gym.

Flaw #6: Unclear Wording

This includes bad grammar and confusing language, anything that makes your questions unclear. In addition, words like “always” or “never” rarely apply to customers’ real circumstances and therefore should be avoided. Edit, proof, and test and then edit, proof, and test again.

Flaw #7: Excessive Use of Required Questions

Too many required questions forces customers to answer questions that don’t apply to them, or that they simply don’t want to answer. This increases survey abandonment and nonsense answers.

Flaw #8: Under OR Overused Text Fields

When your survey does not provide enough text fields for customers to explain their answers, valuable information is lost. Conversely, too many text questions is overwhelming and leaves customers mentally exhausted.

Flaw #9: Jargon

Customers are usually not familiar with your company’s internal language. Using internal terms leads to skewed data as customers may interpret questions in ways you didn’t intend.

Flaw #10: Questions the Customer Can’t Answer

These are questions that are best answered by someone besides your customer. For instance, we’ve seen the question “rate the white space to text balancing”. This is an example of a question best answered by a design team.

Flaw #11: Insufficient Use of Logic Gating

Customers have a limited amount of energy they will spend on your behalf. The longer the survey, the lower the answer accuracy and completion rate. Good surveys use logic to maximize question relevance and minimize survey length.
The Alaska Airlines survey mentioned earlier failed miserably with logic gating. In fact, Alaska Airlines punished frequent fliers by asking even more questions. Don’t overload your surveys. Instead, have a suite of surveys, each one used for a different purpose.

Flaw #12: Copycat Questions

These are the questions like “how likely are you to recommend” that you see everywhere. Copycat questions might have a place in your survey but make sure that’s the case. Presumably, you want your questions to show you are truly listening and interested in gathering specific insights—right?!

Flaw #13: Limited Answer Options

If the answer options don’t cover the full range of scenarios, customers may provide answers even if none of the options apply. If your options are not exhaustive, include “other” or text fields.

 Survey Bias: Scaling that’s Out of Sync 

Scaling flaws happen when there is no clear mid-point to your answer options, or the anchors or selections are off-kilter in some way.

Flaw #14: Vague Scales

These are scales in which customers don’t perceive a clear difference between two options. For example, customers may not see “Exceeded Expectations” and “Above Expectations” as fundamentally different, yet they are measured 1 point (20%) apart on a 5-point scale.

Flaw #15: Unbalanced Scales

These are scales in which the middle value is not the neutral response. It’s important to give your respondents a neutral option so they aren’t forced to give a skewed answer. An advantage of Net Promoter Score with its 0 to 10 scale, is that it has a natural midpoint of 5.

Flaw #16: Non-parallel Scales

These are scales in which responses on the negative end of a scale are not the opposite of responses on the positive end. Be sure to use parallel anchors so that your scales make intuitive sense.

 Survey Bias: Analysis Weaknesses 

The point of a customer survey is to find out what’s going on for your customers on a daily, weekly, or monthly basis and figure out what to change. When there is no team responsible for finding the story in the data, all that data you’ve collected goes to waste. The remedy? Prioritize analysis.

Flaw #17: Calling Hunches Facts

Sometimes differences that seem important could just be due to random chance—a finding which should be disclosed, or simply thrown out.

Flaw#18: Ignoring Variance

When the analysis only looks at averages, not the distribution of scores, you’re failing to get a nuanced picture. Instead, segment your data to determine what kinds of customers score low/high, and for what products, in what situations?

Flaw #19: Nonsense Graphics

This is when data is presented in ways that confuse the issue. Using a pie chart when the answers don’t add up to a whole or comparing unlike phenomena in a single chart are examples of nonsense graphics.

Flaw #20: Wasting Customers’ Comments

Text answers must be tagged, coded, and quantified, otherwise, key insights are lost. Reading text doesn’t work because you won’t get quantification or a visualization of your customers’ comments. Software-based text analytics are a great start. However, for your text analysis to be accurate, it’s essential to use researchers at least once per month to edit that output and test your customers’ comments for new, emergent themes.

Toward a World with Better Surveys

It takes time and effort to learn how to survey well. But by eliminating these 20 flaws, you’ll be on your way to more effective customer surveys–the kind of surveys that enable you to steer your company tactically and strategically.

If you’re not sure where to start, reach out! We’ll examine your survey for free. If it’s good, we’ll tell you! If we find flaws, we’ll let you know what they are. And if you want to talk more generally about customer surveys, that’s good too.

It’s time for companies to get serious about how they collect customer feedback—and for all of us to demand better customer listening! Toward the best customer surveys!

 

Categories: Customer Surveys
E-book cover

Do you know the benefits of CX science? Download now.

  • This field is for validation purposes and should be left unchanged.

5 minute read: Take your CX program to the next level.