Why a survey does not measure what is actually there

2026-07-26

A survey is the safest choice you can make. Everyone can take part, nobody has to speak up in a room, and at the end you have numbers. That feels more objective than an afternoon of discussion.

Yet a questionnaire rarely delivers what you needed. For a reason that is not obvious at first glance.

You measure what you already knew

In a survey the author determines two things: which questions are asked and which answers are possible. Anything outside that cannot surface.

That is fine when you want to test what you already suspect. But it is fatal when the question is: what is actually alive here? The very answer you had not thought of, the answer you ran the study for, is excluded by construction.

A survey answers your questions. It does not tell you which question you should have asked.

Isolated scores say little about coherence

Suppose you have twenty statements rated and five come out with a high average. What do you know then?

You know which items score highly in isolation. You do not know whether those five form one topic or five different ones, whether they reinforce or exclude each other, or whether the group considers them related at all. Yet those are exactly the questions a decision rests on.

An average also hides disagreement. A statement half the group strongly agrees with and half strongly disagrees with gets the same average as one everyone is neutral about. Those are two entirely different situations, and the first is precisely the one that calls for a conversation.

What it is good for

This is not an argument against questionnaires. They are excellent when you have a bounded question with known answer options: how satisfied people are, how often something happens, which of these three options is preferred.

The misunderstanding arises when a survey is used for an exploratory question. It then pretends to measure something it cannot measure.

The alternative: let the group build the structure

In Group Concept Mapping the order is reversed. First, participants supply the content themselves, without predetermined categories. Then each of them groups those same statements in their own way. Only after that is anything scored.

That third point being last is no accident. You are then scoring statements that came from the group itself, organised into themes the group itself recognises.

And the computation does something a survey cannot: it shows which statements participants consider related, which themes are sharply delineated and which are diffuse. That last part is valuable, because a diffuse theme means people understand different things by it. That is not measurement error but a finding, and usually the first one you need to address.

When to choose what

In short: use a questionnaire when you know what you want to know and the answer options are fixed. Use concept mapping when the question is open, when you want to know which themes exist according to those involved, or when support matters for what happens next.

More on how that works is at the method.