Analyzing Open Ended Survey Questions—Is AI Your Solution?

Analyzing open ended survey questions is the single most fruitful method for getting meaningful, honest feedback. Employees and customers can express themselves through text and say what’s really on their minds in a way that’s impossible through structured rating questions.

Open-ends are your ‘gold’ but extracting the gold is challenging, which is why companies look to AI to solve the problem. But is AI your best solution? If so, when and where should it be applied? Are there any downsides? This article answers these questions.

Jump to: How Do ChatGPT, Gemini, and Claude Stack Up for Text Analysis?

Open-Ended vs. Closed-Ended Questions

First, let’s get clear about the differences between open-ended and closed questions.

Examples of closed-ended questions are:

• The Net Promoter question
• Rating questions
• Binary (Yes/No) questions

These types of questions are useful when all possible responses can be captured in a short, simple list.

But open-ends are a better fit when you’re asking about topics that aren’t straightforward or where responses might vary widely. Because respondents aren’t limited by predetermined answers, they can express their most authentic and wide-ranging opinions. This in turn, allows you to uncover answers to questions you might not have even thought to ask about in the ‘main’ parts of your surveys.

Some examples of open-ended questions include:

• What is <Competitor123> is doing better than <CompanyABC>?
• What was the best part of your experience with <CompanyABC>?
• If a colleague asked you about <CompanyABC’s> products, what would you say?

The Challenges of Analyzing Open Ended Survey Questions

Open-ended survey answers are rich with valuable information.

But analyzing the results of your open ended survey questions requires you to synthesize all of that unstructured information into a coherent form.

Simply reading through comments isn’t practical. With so many responses, it’s nearly impossible to process and glean actionable insights. Moreover, our brains are limited by Miller’s law, which states that working memory can only handle about seven items at once. This makes it easy to overlook subtle patterns and emerging themes in the data.

Even worse, you won’t be able to quantify the emergent themes in the comments, so you won’t have a compelling report. After all, business audiences need more than stories; they need metrics and clear priorities.

Some researchers suggest reading only a few comments to get the general gist of things.

Others turn to word clouds because they’re the fastest and easiest way to give shape (no matter how spurious) to your open ends: just input your comments to see which words occur most often.

But of course, word clouds don’t find meaningful themes, and they don’t show what actions to take. For example, perhaps “product” occurs most often in your word cloud.  But that doesn’t tell you whether customers are pleased or displeased with your product or how to improve the product experience.

Beyond Word Clouds

To make sense of open-ended survey data, you need Text Analysis, which, when conducted by experts (that’s us!) uncovers insights you can use. Here are some of the ways Text Analysis adds value:

1. Sentiment: Determines whether responses are positive, negative, neutral, or mixed. For example, a phrase like “I love this product” signals positive sentiment, while “I’m disappointed with the service” points to negative sentiment. Whether you are using AI or human researchers, this is by far the easiest aspect of comments to analyze — that said, because of mixed sentiment, AI often gets this wrong, and the fact is that having conflicting feelings about a company (or its products or services) is very common.
2. Effort: Identifies how easy or difficult it is for users to achieve their goals with your product or service. Comments mentioning “frustrating” or “easy to use” highlight areas where customers struggle or succeed. Reducing customer effort can significantly improve satisfaction and loyalty.
3. Recurring Themes: Highlight broader patterns across responses, allowing you to address problems or amplify areas where you’re strong. TIP: When analyzing open ended survey questions, start by considering which department each comment belongs to, and keep in mind that often within a single comment, multiple departments may be involved.

Text Tagging: The Social Science Technique

Text tagging categorizes comments using a system of definitions and protocols.

This technique is used by Sociologists, Anthropologists, and Psychologists. And it’s useful for analyzing the Customer Experience too!

Often, tagging is the most accurate and efficient method for extracting meaning from survey verbatims.

At a high level, here’s how the tagging process works:

• Several analysts collaborate to build a tagging framework, cross-checking each other’s work until the framework captures the meaning of the text.
• Then, the framework is used to classify customers’ comments, with each classification becoming progressively narrower and more specific.
• Once all the tagging is in place, the tags are quantified to reveal themes in rank-order priority. Quantification is critical because it enables the emergent themes to be prioritized. In addition, quantification allows text themes to be correlated with outcome metrics like Net Promoter and Customer Satisfaction Scores.

For text tagging to be effective, it’s critical to hire researchers who are experienced with this technique and who are familiar with analyzing customer experience nuances–because nothing about customers’ comments is ever straightforward.

Text Analysis: The Details

Customers’ open-ended responses are often written informally, and it requires some work to assign them a category before they can be labeled accurately.

This requires a five-step process:

1. Substantive: Is the comment clear enough to be tagged objectively?
2. Department: What department is the customer referring to? For instance that could be Repairs, Websites, Customer Support, Returns, Marketing, Warranties, etc..
3. Sentiment: Is the customer happy?  Exuberant? Frustrated? Angry?  Etc.
4. Topic: What’s the comment about? This is the most important part of the classification process, but it’s meaningless without going through the first four steps.
5. Sub-Topic: What specific product or service does this comment address?

Most emphatically, the point of our 5-step process is to eliminate subjectivity. When multiple researchers independently arrive at the same conclusions, you know the findings are solid and that you can use this information to set new tactical or strategic directions.

Why Can’t You Just Use AI For This?

You can use AI when analyzing open ended survey questions, and that’s often what companies do, especially when they have large, continuously updated datasets. However, for AI to be accurate, no matter what kind of company you are or what size, for accurate results, you’ll need to start by doing your own Text Analysis using human researchers operating not from an LLM but from general intelligence. Then, once you’ve set the framework up, you’ll need to audit the results your AI is giving you continuously.

Remember, currently, what is called AI is really a large language model (LLM). AI hasn’t yet achieved anything like intelligence or consciousness. Maybe in two years? Perhaps in 20? No one really knows. And without general intelligence, not only are hallucinations possible, but nuances may be entirely overlooked.

Here’s what your auditors need to pay close attention to:

• When sentiment is mixed, AI often gets it wrong by overfocusing on the first sentiment.
• While content extraction is improving, AI extracts meaning per question—but customers tend to repeat themselves among open-ended questions.
• Most AI tools write a summary of the themes, but the best action happens at the customer level.

TIP: Sentiment analysis is the easiest part of verbatim analysis for humans, but we’ve seen AI algorithms with insufficient training incorrectly tag sentiment more than 70% of the time when sentiment is mixed or word choices are confusing. Look how difficult it can be to get sentiment right using AI:

In this example, it’s easy for humans to tell that the sentiment is negative—CompanyABC lost the project to another vendor. However, many AI tools we’ve tested mark it as a positive comment because of the word “award.”

Emotions are complicated, and comments have many cues that AI struggles to understand. And yet, sentiment analysis is only one piece of the puzzle and not the most actionable piece. It’s the themes — the content — in which the power of comments surfaces.

Here’s another example of two customer comments that humans innately understand are different but that AI is likely to treat as the same: “it was okay.” VS. “It was okay, I guess.”

Now consider another issue with AI.

Most AI programs that work at the customer level (not highlighted summaries like what you get from Zoom, Sonix, etc. ) are set up to analyze the answers to each of your open ends. So, if customers repeat themselves, which they often do, AI might overemphasize certain topics—so you’ll wind up with an over-indexing of those themes.

Humans, on the other hand, intuitively recognize when a customer is reinforcing a single point across multiple questions rather than introducing entirely new topics, which helps avoid misinterpreting the data.

For example, consider this customer’s answers to the open-ended questions in CompanyABC’s survey:

AI tools might consider that the side button is more important than it truly is because it was mentioned three times—but in reality, the same customer repeats the side button comment 3 times, so it’s not as important to the group overall, but rather to this individual customer.

More about ChatGPT etc. is below, but the most common LLMS are designed simply to give you a narrative summary of ALL the comments (or answers to interview questions); they don’t provide customer-by-customer summaries, at least at this time.

We use many AI tools. So, we’re not down on AI; we just want you to understand its limitations so you know how to use it effectively.

How Do ChatGPT, Gemini, and Claude Stack Up for Text Analysis?

All three AI tools—ChatGPT, Gemini, and Claude—are capable of analyzing open-ended survey questions and other types of unstructured data. However, each has its unique strengths, limitations, and approaches to organizing insights.

We tested the advanced version of each of the three tools with a real customer dataset to see how they extract themes and summarize the data, and this is what we found. (We will continue to test the three main LLMS using different datasets and will update our findings frequently.)

Theme Extraction and Organization

Each tool derives themes and organizes them differently, often prioritizing certain insights over others:

• ChatGPT: Tends to pull out overarching themes and narratives. For instance, in a customer survey, it might highlight that “customers think the product is too expensive but high quality.” It often provides a more generalized summary, which is useful for creating a high-level understanding of the data.
• Gemini: Extracts themes differently, sometimes prioritizing operational aspects. Using the same dataset, Gemini might identify “the repair process takes too long” as the top theme. This approach can be particularly useful for pinpointing logistical or operational challenges.
• Claude: Organizes results into bullet points, often grouping insights by specific attributes. For example, instead of providing a single top theme, Claude might generate a list such as:
  • Products perceived as too expensive: Product A, Product B
  • Issues with repair time: Product C, Product D

This detailed categorization can be valuable when granular insights are required to address specific areas of concern.

Strengths and Use Cases

• ChatGPT: Best for crafting a narrative or summary of customer sentiment and key themes. It excels at producing well-written overviews and is ideal for presentations or reports requiring a topline narrative.
• Gemini: Offers a more action-oriented approach by highlighting operational themes that may require immediate attention. It’s particularly useful for businesses looking to address process-related issues.
• Claude: Delivers highly structured results that allow for a more analytical, itemized view of the data. It’s a strong choice for dissecting complex datasets with multiple dimensions.

Drawbacks and Limitations

• ChatGPT: May overgeneralize themes, potentially missing subtle nuances. It requires careful prompting to ensure themes are prioritized correctly and not overlooked due to a lack of specificity.
• Gemini: While prioritizing operational issues can be helpful, it may underemphasize emotional or subjective customer sentiments, leading to a more mechanical interpretation of the data.
• Claude: Its bullet-point style and categorization might lack the cohesive narrative some users prefer. Additionally, it can sometimes focus too heavily on granular details, making it harder to see overarching trends.

Finding the Right Fit for Your Needs

When analyzing open-ended survey questions, choosing between ChatGPT, Gemini, and Claude depends on your specific goals:

• Need a high-level narrative for presentations? Go with ChatGPT.

• Need actionable insights into operational challenges? Try Gemini.
• Need a detailed breakdown of specific issues? Claude is your best bet.

AI and Analyzing Open Ended Survey Questions

With the prevalence of LLMs, AI is finding new uses at a rapid rate. Is analyzing open ended survey questions one of them?

AI might be your solution if:

• You have researchers to set it up.
• Your dataset tends to be large and repetitive.
• You’re willing to audit the results periodically.

AI is NOT a good solution if:

• You have limited data: Midsize or smaller organizations may not have enough data for AI to perform accurately.
• You’re working with isolated datasets: AI isn’t practical for one-time surveys or occasional tracking studies due to the high upfront training costs.
• You use specialized terminology: AI requires extensive training to understand niche-specific terms, acronyms, or codes.
• You have budget constraints: AI-driven text analysis can be expensive; manual tagging by a research team might deliver similar results at a fraction of the cost.

So, is using AI for open ended question analysis a good fit for you? It could be, but you need to know how to use it so you can decide for yourself whether it’s the right choice for your company.

Here’s the truth: All AI-driven text analysis solutions begin with the same five-step process of researchers tagging comments.

Why? Because, for AI to be accurate, it must be trained. Who does the training? A team of researchers. And how do they train the algorithm? By tagging comments using these very same classic social science techniques.

So, it’s not a question of either Text Analysis or AI, because AI starts with Text Analysis. The question is whether your dataset is big enough and uniform enough to train an AI model. Either way, you’ll start with a team of researchers tagging!

Key Takeaways

• Open-ended questions allow your customers to share their thoughts, offering deeper insights than structured questions. They reveal themes you didn’t think to ask about.
• Analyzing open ended survey questions is challenging due to the volume and complexity of data, but Text Analysis offers structured methods to extract actionable insights.
• AI tools like ChatGPT, Gemini, and Claude excel at summarizing data, identifying themes, and speeding up the analysis process—but each has limitations—so you still need human researchers.
• AI might not always be suitable for open ended question analysis unless your dataset is large and continuously updated.
• Manual Text Analysis might be a better fit for smaller datasets, isolated surveys, or specialized terminology.
• When AI is used for analyzing open ended survey questions, it still needs training based on Text Analysis methods.

Text Analysis: A Proven Method

Using Text Analysis to extract the gold in your open-ended survey questions is a proven way to unlock critical business insights.

============================================

Care to discuss Text Analysis? Get in touch!

============================================