How to analyse open-ended survey responses

The most useful question on most surveys is the one with a blank box under it: "Is there anything else you'd like to tell us?" It is also the one most teams never properly analyse. The ratings go into a chart; the comments go into a spreadsheet tab that someone scrolls through once and forgets.

That is a waste. Open-ended answers are where customers tell you why a score is what it is, and where they raise things you never thought to ask about. This guide sets out a method for turning a pile of free-text replies into something you can count, compare and act on, whether you do it by hand or with software.

What you are trying to produce

Before you start, be clear about the output. Analysing open-ended responses well means ending up with three things:

1. A list of topics that customers talk about, organised so you can see both the big picture and the detail.
2. A count for each topic, expressed as the share of replies that mention it, so periods and groups of different sizes can be compared.
3. A tone for each mention, so you know whether people are praising a topic or complaining about it.

Everything else (charts, reports, decisions) is built from those three. If you finish with a word cloud and a handful of favourite quotes, you have not analysed the replies; you have sampled them.

Step 1: Read before you code

Take a random sample of fifty to a hundred replies and read them all, slowly, before you name a single topic. Resist the urge to start labelling on reply one. Early replies set your expectations, and if you code as you go, the first ten answers shape the whole scheme.

As you read, jot down the things people mention in plain words: "waited ages at reception", "kids loved the pool", "app logged me out". Don't group yet. You are collecting raw material.

Step 2: Build a codebook as a topic tree

A codebook is the list of topics you will tag replies with, plus a one-line definition of each. The most practical shape for it is a tree: broad topics at the top, specific ones beneath.

For a hypothetical 40-room hotel, part of the tree might look like this:

• Arrival
  • Check-in speed
  • Welcome and staff manner
  • Parking
• Room
  • Cleanliness
  • Bed and sleep
  • Noise
• Food and drink
  • Breakfast
    • Coffee
    • Opening hours
  • Bar

Nesting matters for two reasons. First, it lets you report at whatever level the reader needs: the general manager wants "Food and drink"; the breakfast chef wants "Coffee". Second, it gives new, unexpected comments somewhere sensible to land. A reply about "no oat milk" fits under Breakfast even if nobody has mentioned it before.

Good codebook rules:

• Name topics after the thing, not the feeling. "Check-in speed", not "Slow check-in". Tone is recorded separately, so the same topic can hold praise and complaints.
• Write a definition for each. One sentence saying what belongs there and what doesn't. "Noise: sounds that disturbed sleep or rest, from inside or outside the building. Not music in the bar, which goes under Bar."
• Keep a catch-all, and empty it often. An "Other" bucket is fine as long as you review it. When three or more replies in it share a theme, give that theme its own topic.
• Start broad, split later. It is easier to split a big topic into two than to merge ten tiny ones that turn out to be the same thing.

Step 3: Code every mention, not every reply

One reply usually mentions several things. "Room was lovely and quiet but breakfast ran out of eggs by 9" touches Cleanliness or Room in general, Noise and Breakfast. Tag all of them.

If you force each reply into a single topic, you will under-count everything that tends to appear second in a sentence, which is often the complaint.

Alongside each topic, record the tone of that mention: positive, negative, mixed or neutral. In the example, Noise is positive and Breakfast is negative. Our guide to customer feedback sentiment analysis goes into why tone belongs on the mention rather than on the reply as a whole.

Step 4: Keep coding consistent

The main weakness of manual coding is drift. The same person codes differently on Monday morning and Friday afternoon; two people code differently from the start. A few habits help:

• Double-code a sample. Have two people code the same thirty replies independently, then compare. Where they disagree, the definition is unclear; rewrite it.
• Log decisions. When you decide that "the shower was cold" goes under Room → Bathroom rather than Maintenance, write it down so next month's coder makes the same call.
• Re-check old replies after a split. If you split Breakfast into Coffee and Opening hours, go back and re-file the earlier Breakfast mentions, or your trend will show a sudden jump that is really just a change of labels.

Manual coding versus AI

Coding by hand is entirely workable for a few dozen replies a month. It forces you to read every word, which has real value. The trouble starts in the hundreds: it becomes a part-time job, the backlog grows, and consistency slips.

Language models are now good at this particular task. Given a topic tree, they can file each mention into it and give it a tone, quickly and the same way every time. They are also good at proposing topics from a fresh batch of replies, which saves the blank-page stage.

The honest trade-offs:

The last row is the real risk with automation: it becomes possible to never read a reply again. Don't let it.

Step 5: Check the machine's work

Whether a person or a model did the coding, check it the same way.

• Spot-check each big topic. Open ten replies filed under it. Do they belong? Is the tone right?
• Read the low-confidence edges. Short, sarcastic or multi-language replies are where any coder, human or machine, goes wrong most.
• Look at what was left out. Read a handful of replies that ended up with no topic or only in Other. Is something new hiding there?
• Follow every number back to replies. If a chart says complaints about parking rose, you should be able to click through, or at least filter, to the replies behind it. A figure you cannot trace is a figure you cannot trust.

Step 6: Count by share, then compare

Once replies are coded, count each topic as a share of replies: the percentage of all replies in the period that mention it. Raw counts mislead whenever the number of replies changes, and it nearly always does. A busy month will make every topic look bigger.

Then compare against the previous period. The interesting findings are almost always changes:

• A topic whose share grew noticeably.
• A topic whose tone turned worse, even if its share stayed flat.
• A topic that appeared for the first time.

Rank the negative ones by how many customers they touch, how strongly they are linked to low scores, and whether they are getting worse. Then go back and read the replies for your top three. The counts show you where to look; the words show you what is actually happening.

Step 7: Share it in a form people use

Analysis that stays in a spreadsheet changes nothing. Write the findings up briefly: the short answer first, then each finding with a number, a chart if it helps and one or two real replies quoted. Our template for a monthly customer feedback report shows one way to lay it out.

And if your open-ended answers are thin ("fine", "ok", "n/a"), the problem may be the question rather than the analysis. See how to write a feedback form people finish for ways to ask that draw out useful detail.

Doing this with Userforms

Userforms reads every reply as it arrives, whether it comes from one of your forms, a CSV you upload, pasted text or imported reviews, and files each mention into a topic hierarchy with its own tone. You can rename, merge and split topics as your picture sharpens, and the dashboard compares each topic's share of conversation and tone with the previous period.

Every chart opens onto the replies behind it, so checking the work is a click rather than a search. You can also ask plain-language questions of your replies and see which replies each answer relied on.

You can start for free, or compare plans on the pricing page.