This morning, my rep texted me from the shop where my car is serviced:
“You’ll receive a survey from us. It would mean a lot to me if you completed it. We hope you recommend us.”
I get he wants a positive review, but this is an example of clear survey bias.
Essentially, my rep is pushing for the answers he wants his managers to see.
This is not scientific data collection. It’s public relations or marketing. Either way, it’s what we call Gaming and it’s the second flaw on the survey bias list below.
I frequently interact with my service rep, so I have everything to gain by giving him a good score.
Since all I care about is getting the lowest fees, it’s simply not in my best interest to provide the most honest feedback. Basically, even if my rep is rude, I’ll give him a top score to stay on his good side for the best pricing.
Is the point of the survey for customers to get good pricing?
Or, is it for the company to measure the quality of their customer service?
Why Issue Surveys?
Why ask your customers to take surveys if the data you collect doesn’t capture real facts?
Bad Customer Surveys are Ubiquitous
Earlier this year I wrote about how Whole Food’s point of purchase receipt is a complicated, disorganized mess.
I also explored how Alaska Airlines is pestering its customers with surveys that ask more than 94 questions.
Moreover, in 2016, we conducted a study of the surveys issued by the nations’ largest retailers. With one exception (7-Eleven), the surveys by the top retailers were unscientific and failed the basics of customer engagement.
Basically, bad customer surveys are not only ubiquitous, they’re a multi-billion-dollar industry!
Let’s Imagine a World with Good Surveys
With this goal in mind, I have compiled a checklist so you can vet your own surveys for survey bias. There are 4 main categories of survey bias, within which there are 20 types of flaws.
Survey Bias: Bad Data from the Start
Have you ever had a contractor bring a washing machine to your house and show you a card that says,
“Please give me 5 stars my job depends on it!”?
Or maybe you bought a big-ticket item and were told they could come down in price for a good survey score? Garbage in. Garbage out. If you think your company may be collecting bad data, use mystery shoppers to document which customers are asked to take your survey — and how.
There are two main ways data can be bad from the start.
Flaw #1: Selection Bias
This is when your survey data is drawn from misrepresentative samples of your customer base and doesn’t represent your customers at large. For example, your data could come from those with lots of free time, those with a specific gripe, or those who gave their email addresses.
Basically, with misrepresentative samples, your survey omits one or more customer groups—or your survey sample is too small to give you reliable facts.
Flaw #2: Gaming
This is when associates only survey customers they believe had positive experiences. They also might ask customers to answer the survey in exchange for implied or explicit favors.
Survey Bias: Poor Question Wording
How questions are phrased has everything to do with whether you will get accurate data and whether your customers will engage with your survey. The best way to clean up wording flaws is to work with an objective team, outside of your company. But if that’s impossible, at least work with another department.
There are 11 ways survey wording can be flawed.
Flaw #3: Leading Questions & Statements
These questions prompt customers for the answers you want to hear. Both tone (when done over the phone) and content are used to push customers toward particular answers.
For example, “How satisfied were you?” assumes the customer was somewhat satisfied.
Flaw #4: Double-Barreled Questions
These questions ask two things at once, so it’s unclear which question the customer is answering. For example, “Was your rep efficient and proactive?” asks about two different behaviors in one question.
Flaw #5: Forced Questions
These questions ask your customers to say something about themselves that may not be true. For example, “What’s your favorite gym?” when the customer may not use a gym.
Flaw #6: Unclear Wording
This includes bad grammar and confusing language, anything that makes your questions unclear. Edit, proof, and test and then edit, proof, and test again.
Flaw #7: Excessive Use of Required Questions
Too many required questions forces customers to answer questions that don’t apply to them, or that they simply don’t want to answer. This increases survey abandonment and nonsense answers.
Flaw #8: Under OR Overused Text Fields
When your survey does not provide enough text fields for customers to explain their answers, valuable information is lost. Conversely, too many text questions is overwhelming and leaves customers mentally exhausted.
Flaw #9: Jargon
Customers are usually not familiar with your company’s internal language. Using words and phrases that are only familiar to you leads to skewed data because customers may interpret questions in ways you didn’t intend.
Flaw #10: Questions the Customer Can’t Answer
These are questions that are best answered by someone besides your customer. For instance, we’ve seen the question “rate the white space to text balancing”. This is an example of a question best answered by a design team.
Flaw #11: Insufficient Use of Logic Gating
Customers have a limited amount of energy they will spend on your behalf. The longer the survey, the lower the answer accuracy and completion rate.
Good surveys use logic to maximize question relevance and minimize survey length.
The Alaska Airlines survey mentioned earlier failed miserably with logic gating. In fact, Alaska Airlines punished frequent fliers by asking them the most questions.
Don’t overload your surveys.
Instead, have a suite of surveys, each one used for a different purpose.
Flaw #12: Copycat Questions
These are the questions like “how likely are you to recommend” that you see everywhere.
Copycat questions might have a place in your survey but make sure that’s the case. Presumably, you want your questions to show you are truly listening and interested in gathering insights—right?!
Flaw #13: Limited Answer Options
If your answer options don’t cover the full range of scenarios customers encounter, customers may check any random box on your list, even if it has nothing to do with their situation. If for some reason your answer options are not a complete list, then include “other” or text fields.
Survey Bias: Out of Sync Scaling
Scaling flaws happen when there is no clear mid-point to your answer options, or the anchors or selections are off-kilter in some way. These are the three most common scaling flaws.
Flaw #14: Vague Scales
These are scales in which customers don’t perceive a clear difference between two options.
For example, customers may not see “Exceeded Expectations” and “Above Expectations” as fundamentally different, yet they are measured 1 point (20%) apart on a 5-point scale.
Flaw #15: Unbalanced Scales
These are scales in which the middle value is not the neutral response. It’s important to give your respondents a neutral option so they aren’t forced to give a skewed answer.
An advantage of Net Promoter Score with its 0 to 10 scale, is that it has a natural midpoint of 5.
Flaw #16: Non-parallel Scales
These are scales in which responses on the negative end of a scale are not the opposite of responses on the positive end. Be sure to use parallel anchors so that your scales make intuitive sense.
Survey Bias: Flawed Analysis
The point of a customer survey is to find out what’s going on for your customers on a daily, weekly, or monthly basis–and from that data, figure out what to change.
When there is no team responsible for finding the story in the data, your survey effort goes to waste. The obvious remedy is to prioritize analysis.
There are many ways analysis can be flawed, but here are four of the most glaring analysis mistakes.
Flaw #17: Calling Hunches Facts
Sometimes differences that seem important could just be due to random chance—a finding which should be disclosed, or simply thrown out.
Flaw#18: Ignoring Variance
When the analysis only looks at averages, not the distribution of scores, you’re failing to get a nuanced picture.
Instead, segment your data to determine what kinds of customers score low/high, and for what products, in what situations?
Flaw #19: Nonsense Graphics
This is when data is presented in ways that confuse the issue. Using a pie chart when the answers don’t add up to a whole or comparing unlike phenomena in a single chart are examples of nonsense graphics.
Flaw #20: Wasting Customers’ Comments
Text answers must be tagged, coded, and quantified, otherwise, key insights are lost. Reading text doesn’t work because you won’t get quantification or a visualization of your customers’ comments.
Software-based (AI) text analytics are a great start. However, for your text analysis to be accurate, it’s essential to use researchers at least once per month to edit that output and test your customers’ comments for new, emergent themes.
Toward a World with Better Surveys
It takes time and effort to learn how to do a survey well. But by eliminating these 20 flaws, you’ll be on your way to more effective customer surveys–the kind of surveys that enable you to steer your company tactically and strategically.
It’s time for companies to get serious about how they collect customer feedback—and for all of us to demand better surveys!
May the best surveys win!
================================================
Interaction Metrics builds scientific surveys that result in decisive outputs and actions. If you’re not sure where to start, reach out! We’ll examine your survey for free. If it’s good, we’ll tell you! If we find flaws, we’ll let you know what they are. And if you want to talk more generally about the customer experience, we can do that, too.
================================================
