This morning, my rep texted me from the shop where my car is serviced:
“You’ll receive a survey from us. It would mean a lot to me if you completed it. We hope you recommend us.” I get what he wants, but this is NOT how to do a survey.
Essentially, my rep is pushing for the answers he wants his higher ups to see. But that’s not scientific data collection. It’s public relations or marketing. Either way, it’s what we call Gaming and it’s the second flaw on the list below.
Since I frequently negotiate with my rep, I have nothing to gain by taking this survey truthfully. Basically, by giving my rep a good score, I am more likely to win on future fees, and that’s my main concern. It doesn’t matter that I find my rep rude, I’ll give him a top score to stay on his good side and get the best pricing.
But is the point of the survey for customers to get good pricing? Or is it for the company to measure the quality of their customer service?
Why Issue Surveys?
Why ask your customers to take surveys if the data you collect fails to capture the facts? Even worse, why issue surveys if the net result could be that your customers appear happier than they are.
Bad Customer Surveys are Ubiquitous
Earlier this year I wrote about how Whole Food’s point of purchase receipt suffers from a complicated, disorganized survey request. I also explored how Alaska Airlines is pestering its customers with surveys that extend to 94 (!) questions.
Moreover, in 2016, we conducted a study of the surveys issued by the nations’ largest retailers. With one exception (7-Eleven), the surveys by the top retailers were unscientific and failed the basics of customer engagement. Learn more about the study and the criteria we used to evaluate the surveys here.
Basically, bad customer surveys are everywhere. In fact, they appear to be a multi-billion-dollar industry.
Imagine a World with Better Surveys
Let’s imagine a world in which companies ask meaningful questions in meaningful ways. Companies would only use surveys to interact authentically and learn from their customers. They would put a kibosh on leading questions and never bug customers with irrelevant or rote inquiries.
Toward this end, I have compiled some of the checks we use to vet the customer feedback programs that we manage for our clients. I divided this list of 20 flaws into 4 main types of survey errors. Let’s get started!
Bad Data from the Start
Have you ever had a contractor bring a washing machine to your house and show you a card that says, “Please give me 5 stars my job depends on it!”? Or maybe you bought a big-ticket item and were told they could come down in price for a good review? Garbage in. Garbage out. If you think your company may be collecting flawed data, develop a ‘how to survey’ strategy. Start by using mystery shoppers to document which customers are asked to take your survey — and how.
Survey Bias #1: Mis-Representative Samples
This is when your survey data only comes from certain kinds of customers and doesn’t represent your customers at large. For example, your data could come from those with lots of free time, those with a specific gripe, or those who gave their email addresses. Basically, with misrepresentative samples, your survey sample omits one or more customer groups—or your survey sample is too small to give you reliable facts you can count on.
Survey Bias #2: Gaming
This is when associates only survey customers they believe had positive experiences. They also might ask customers to answer the survey in exchange for implied or explicit favors.
Poor Question Wording
How questions are phrased has everything to do with whether you will get accurate data and whether your customers will engage with your survey. The best way to clean up wording flaws is to work with an objective team, ideally outside your company. But if that’s not possible, at least work with another department.
Survey Bias #3: Leading Questions & Statements
These questions prompt customers for the answers you want to hear. Both tone (when done over the phone) and content are used to push customers toward particular answers. For example, “How satisfied were you?” assumes the customer was somewhat satisfied.
Survey Bias #4: Double-Barreled Questions
These questions ask two things at once, so it’s unclear which question the customer is answering. For example, “Was your rep efficient and proactive?” asks about two different behaviors in one question.
Survey Bias #5: Forced Questions
These questions ask your customers to say something about themselves that may not be true. For example, “What’s your favorite gym?” when the customer may not use a gym.
Survey Bias #6: Unclear Wording
This includes bad grammar and confusing language, anything that makes your questions unclear. In addition, words like “always” or “never” rarely apply to customers’ real circumstances and therefore should be avoided. Edit, proof, and test and then edit, proof, and test again.
Survey Bias #7: Excessive Use of Required Questions
Too many required questions forces customers to answer questions that don’t apply to them, or that they simply don’t want to answer. This increases survey abandonment and nonsense answers.
Survey Bias #8: Under OR Overused Text Fields
When your survey does not provide enough text fields for customers to explain their answers, valuable information is lost. Conversely, too many text questions is overwhelming and leaves customers mentally exhausted.
Survey Bias #9: Jargon
Customers are usually not familiar with your company’s internal language. Using internal terms leads to skewed data as customers may interpret questions in ways you didn’t intend.
Survey Bias #10: Questions the Customer Can’t Answer
These are questions that are best answered by someone besides your customer. For instance, we’ve seen the question “rate the white space to text balancing”. This is an example of a question best answered by a design team.
Survey Bias #11: Insufficient Use of Logic Gating
Customers have a limited amount of energy they will spend on your behalf. The longer the survey, the lower the answer accuracy and completion rate. Good surveys use logic to maximize question relevance and minimize survey length.
The Alaska Airlines survey mentioned earlier failed miserably with logic gating. In fact, Alaska Airlines punished frequent fliers by asking even more questions. Don’t overload your surveys. Instead, have a suite of surveys, each one used for a different purpose.
Survey Bias #12: Copycat Questions
These are the questions like “how likely are you to recommend” that you see everywhere. Copycat questions might have a place in your survey but make sure that’s the case. Presumably, you want your questions to show you are truly listening and interested in gathering specific insights—right?!
Survey Bias #13: Limited Answer Options
If the answer options don’t cover the full range of scenarios, customers may provide answers even if none of the options apply. If your options are not exhaustive, include “other” or text fields.
Scaling that’s Out of Sync
Scaling flaws happen when there is no clear mid-point to your answer options, or the anchors or selections are off-kilter in some way. Scales matter. Give them their due!
Survey Bias #14 Vague Scales
These are scales in which customers don’t perceive a clear difference between two options. For example, customers may not see “Exceeded Expectations” and “Above Expectations” as fundamentally different, yet they are measured 1 point (20%) apart on a 5-point scale.
Survey Bias #15: Unbalanced Scales
These are scales in which the middle value is not the neutral response. It’s important to give your respondents a neutral option so they aren’t forced to give a skewed answer. An advantage of Net Promoter Score with its 0 to 10 scale, is that it has a natural midpoint of 5.
Survey Bias #16: Non-parallel Scales
These are scales in which responses on the negative end of a scale are not the opposite of responses on the positive end. Be sure to use parallel anchors so that your scales make intuitive sense.
Analysis Weaknesses
The point of a customer survey is to find out what’s going on for your customers on a daily, weekly, or monthly basis and figure out what to change. When there is no team responsible for finding the story in the data, all that data you’ve collected goes to waste. The remedy? Prioritize analysis.
Survey Bias #17: Calling Hunches Facts
Sometimes differences that seem important could just be due to random chance—a finding which should be disclosed, or simply thrown out.
Survey Bias #18: Ignoring Variance
When the analysis only looks at averages, not the distribution of scores, you’re failing to get a nuanced picture. Instead, segment your data to determine what kinds of customers score low/high, and for what products, in what situations?
Survey Bias #19: Nonsense Graphics
This is when data is presented in ways that confuse the issue. Using a pie chart when the answers don’t add up to a whole or comparing unlike phenomena in a single chart are examples of nonsense graphics.
Survey Bias #20: Wasting Customers’ Comments
Text answers must be tagged, coded, and quantified, otherwise, key insights are lost. Reading verbatims doesn’t work because you won’t get quantification or a visualization of your customers’ comments. Software-based text analytics are a great start. However, for your verbatim analysis to be accurate, it’s essential to use researchers at least once per month to edit that output and test your customers’ verbatims for emerging words and themes.
How to Survey
It takes time and effort to learn how to survey well. But by eliminating these 20 flaws, you’ll be on your way to more effective customer surveys–the kind of surveys that enable you to steer your company tactically and strategically.
If you’re not sure where to start, reach out! We’ll examine your survey for free. If it’s good, we’ll tell you! If we find flaws, we’ll let you know what they are. And if you want to talk more generally about customer surveys and the customer experience, reach out!
It’s time for companies to get serious about how they collect customer feedback—and for all of us to demand better customer listening! Toward the best customer surveys!
