About a year ago I sat through the readout for a customer survey we'd commissioned. Three hundred respondents, demographics handled properly, the analysis tidy enough. The recommended decision came out one way, and we made it that way because surveys are what we've got. Nine months later the telemetry was unambiguous and the survey had been wrong from the start.

That isn't an unusual story. Anyone who has worked across marketing, product or analytics for any length of time has a stack of them. What's worth noticing is that everyone in the room had a sense the survey might be wrong, and we used it anyway. We didn't have anything else to base the call on. The asking was the asset.

The polling story isn't what most people remember

The 2020 US election polls produced the largest national polling error in forty years. The American Association for Public Opinion Research published its task force report on it in 2021, and it's worth reading because the conclusions are not what most people came away believing.

The polls underestimated Trump's national support by about 3.3 points, and got the margin wrong by 3.9. The comfortable explanations (shy Trump voters, late-deciders, education weighting, electorate composition) were systematically tested and largely ruled out. What the AAPOR report ends up admitting is something closer to a confession: we don't fully know what went wrong, and the standard explanations don't survive the data.

That finding is more interesting than "polls were bad in 2020". It says the leading professional body in the field, after a year of investigation, couldn't identify a clean methodological lever that would have produced the right answer. And the trend underneath is even less encouraging. Pew Research Center's typical telephone-poll response rate was 36% in 1997 and 6% in 2018. The tool used to read public opinion has lost most of its yield in twenty years.

The thing pollsters now point to most often is nonresponse bias, and it's worth being precise about what that means. The people who don't answer the phone, or hang up, or refuse the survey are not a polite, randomly-selected slice of the people who do. They differ — in age, in trust of institutions, in political leaning, in willingness to engage with strangers. So a 6% response rate doesn't give you a small noisy sample of the population. It gives you a complete sample of an unrepresentative subgroup, with the bias baked in directionally rather than washing out across the average. That's a different problem from "we need a bigger sample". It's a problem that gets worse the harder you push for participation.

The older problem nobody fixed

Polling is the visible failure because elections produce a public scoreboard. Whether the polling case maps cleanly onto focus groups and customer research is a fair question, and I'm not sure it does perfectly — the mechanisms aren't identical, and the social texture of asking someone about a product is different from asking them about a candidate. But the underlying flaw runs through everything built on the same machinery — focus groups, customer satisfaction scores, NPS panels, employee engagement surveys, brand trackers, conjoint studies — and nobody publishes a post-mortem when a survey-led product decision underdelivers.

The deeper issue is what happens inside the head of the person who does respond.

Behavioural economists have the cleaner name for it: constructed preferences. When you ask someone what they want, three things tend to be true at the same time. They haven't thought deeply about it. They feel obligated to produce an answer because you've asked. And the act of answering shapes the answer they end up giving. Add the social layer on top of that — wanting to look thoughtful, deferring to the dominant voice in the room, being polite to the bakery owner asking about her own pastry — and you've got a process that produces a different signal from the one you wanted to measure. It doesn't reveal a stable underlying view. It manufactures one in the moment of being asked. The data isn't noisy. It's the wrong shape of data.

A friend of mine ran focus groups for a major British retailer in the 2010s. She told me, only half-joking, that her job was getting good at predicting which of the things people said they wanted in the focus group they would actually buy when they got home. Her job, properly described, was reading distortion, not eliminating it. She thought she was reasonably good at the reading. She was less convinced the people upstairs understood that the reading was what they were paying for.

This is the part of the customer-input problem I've been chewing on for years, because the instrument turns out to be least reliable in exactly the situations where you most need a clear answer — high-stakes decisions, novel categories, anything where the customer hasn't already developed a settled view.

What changed isn't the asking — it's everything underneath

The reason this matters now, in a way it didn't really matter ten years ago, is that the cost structure has flipped under it.

For most of the time the survey-and-focus-group industry has been running, asking was the cheap option because building was expensive. Putting a real product in front of real customers cost manufacturing tooling, distribution, retail space and six to eighteen months of engineering. Putting twelve people in a room with a moderator and a one-way mirror cost a few thousand pounds and a fortnight. The whole logic — ask, analyse, decide, build — assumed building was the bottleneck.

That assumption isn't true any more, and the people who feel it most keenly are the ones doing the building. I can prototype a software product in a weekend now that would have taken a six-month sprint when I started in marketing. A small team can ship five variants of an offer, a landing page, an onboarding flow and an email sequence, and watch which ones get used, in less time than it would take to recruit and brief a focus group on the same question. Focus groups haven't got worse. The cheap-and-fast alternative now exists where it didn't, and that changes which trade-off makes sense.

The new loop is build, observe, adapt. The most concrete version I've seen is teams who would have spent two or three months on a market-research study getting a cleaner answer in a couple of weeks, by shipping four lightweight feature stubs gated behind "interested in this?" clicks in their main product and watching which ones got pressed. The clicks tend to rank the features differently from any survey that's been run for those customers the year before. That gap is the whole argument.

So the question reverses: why are we still doing the asking-heavy version?

The reason, mostly, is that organisations are slow, and the people whose careers were built on the old loop are the ones who'd need to dismantle it. That's a pattern I've written about in marketing too — the technology shifts faster than the org chart, and the gap between what's now possible and what's actually being done widens for a few years before anyone admits it.

Where asking is still the right call

The next move would be to declare asking dead, and I don't think it's that. There are situations where it's the only honest tool.

Anything that turns on what someone has decided rather than what they're doing (strategic intent, hiring, regulatory consultation) depends on asking. Early-stage research, where you don't yet know what to build and don't yet know what to put in front of people, belongs in interviews and ethnographic work, not telemetry. Anything involving lived experience, particularly in policy and healthcare, is more dishonest if you try to infer it from behavioural traces than if you spend an hour with someone and listen.

What changes is the weight you put on the answer afterwards. It should match what kind of thing the answer actually is, and most organisations weight opinion-style data far more heavily than the underlying mechanism justifies.

Walking back the last decision

A useful exercise, before commissioning anything new, is to walk through the last big decision you made on the back of an opinion-style input (a survey, a focus group, a customer feedback session) and ask what behavioural evidence sat alongside it. Not what the customers said they would do, but what customers like them, or those same customers, had actually done in similar situations, and what the telemetry was already showing about which features got used, which emails got opened, which prices got accepted, which workflows people abandoned three steps in.

If the answer is "not much, mostly the survey", you've been making a confident call on weak signal. That's the system working as designed. The system was designed when you couldn't afford to do better.

It helps to be concrete about what counts as which kind of signal. Someone telling you they'd buy your product is at the bottom rung, contaminated by all the things we've been through. Someone clicking on it when they don't know they're being watched is one rung up. Someone paying for it, using it three months later and bringing it up unprompted with a colleague is the thing you'd actually like to be measuring. The further up that scale the signal sits, the harder it gets to produce in advance — which is precisely why decisions still mostly get made on the bottom rung, and why organisations end up treating "this many people said they'd be interested" as if it were "this many people did the thing".

What I'm less sure about

Honestly, I don't know how fast the rest of this catches up, and I don't think anyone really does. Whole industries (market research, opinion polling, brand tracking) exist because of the old asymmetry, and large institutions don't reorganise around a cost-equation flip in eighteen months. Some of the distortions might be partly correctable. I can imagine smarter weighting, hybrid methods that combine self-report with passive behavioural data, panels that produce more honest answers because the participants are doing rather than talking. None of that has matured yet, and the people who'd build it have an incentive not to disrupt the model that pays them today.

The instrument you've been using is fine, as far as it goes. The cost of using a better one has just come down enough that there's no good excuse left for not moving some of the weight across to it. The companies that get this wrong won't notice they're getting it wrong, because the surveys will keep coming back with answers, and the answers will keep looking like data — until the launch flops and someone has to explain why the customers in the boardroom slide deck weren't the customers in real life.