Validation surveys mislead teams because they measure what people think they want, not what they’ll actually do. Customers give polite, aspirational answers, and teams take them as proof. A well-designed survey gives you useful early signals. Designed badly, it confirms your biases with numbers that look convincing in a presentation.
Surveys feel safe. They’re quick to set up, cheap to run and produce plenty of data that stakeholders trust. That’s exactly what makes them dangerous.
What is a validation survey?
A validation survey is a structured questionnaire that tests specific assumptions about your customers: what they’re trying to get done, what frustrates them and what they’d value. We call it a Validation Survey.
Done properly, it gives you early signals about whether you’re solving a real problem. Done badly, it confirms what you already believe.
The difference is in how you ask.
How a fintech team got 94% positive feedback on a product nobody used
A venture team we were working with inside a bank was building an AI-powered budgeting assistant for millennials. Their surveys showed strong demand for everything they’d assumed.
What they asked:
“How important is automated budgeting to you?” 87% said “very important”.
“Would you use an AI assistant to help manage your finances?” 94% said “yes”.
“How much would you pay for smarter financial insights?” The average answer was £12 a month.
“Rate these features from 1 to 10 for importance.” Every feature averaged 7 or more.
With 94% positive responses and a clear £12 a month price point, they started building a comprehensive budgeting platform. The surveys seemed to confirm everything: the problem, the solution and the price.
Then they soft-launched to the same people who had answered the surveys. Only 12% downloaded the app. Of those, 89% never finished setting it up. The few who did used it twice on average, then gave up.
What their surveys actually measured:
How important people think financial topics should be
Polite answers to questions about AI, which sounds innovative
What people aspire to spend on self-improvement
Feature ratings with no trade-offs involved
What they did instead
Interviews with real users told a different story. People already had budgeting methods that worked “well enough”. They wanted peace of mind about money, not more financial admin. The AI assistant felt like homework, not help.
So the team pivoted from budgeting help to “worry-free” monitoring: alerts when something unusual happened, rather than daily budget tracking. Simple notifications that react to problems, not planning tools that demand effort.
Is your business case built on survey results?
If your evidence for a new idea leans on survey responses, it’s worth a conversation about how to test what customers actually do. Miles, Managing Partner at Future Foundry, can walk through whether a validation survey, or another business experimentation method, fits your situation.
Book a call with Miles here.
Why validation surveys mislead teams
Surveys aren’t flawed in themselves. The problem is that most teams design them to confirm what they hope is true, rather than challenge what might be false. Five problems come up again and again.
1. People want to look good
People give answers that make them seem responsible, forward-thinking and sensible. Nobody admits they don’t want to manage their money better.
2. Hypothetical answers aren’t real behaviour
Surveys ask what people think they would do, not what they actually do when faced with real choices, limited time and competing priorities.
3. There are no trade-offs
Rating features one by one doesn’t show what people choose when they have to pick between options or spend limited time and attention.
4. Questions lead people to the answer
Most teams unconsciously write questions that nudge people towards the answers they want to hear.
5. The “why” goes missing
Surveys capture what people say they want, but miss the situations, constraints and goals that drive what they actually do.
How to write validation survey questions that reveal the truth
The answer isn’t to abandon surveys. Design them like behavioural experiments rather than opinion polls. Ask about what people actually do when faced with real constraints and trade-offs, not what they think they want.
1. Ask about specific past behaviour, not general preferences
Instead of: “Would you use automated budgeting?”
Ask: “When you overspent last month, what did you do the next day?”
What people did in the past best guides what they’ll do next.
2. Force trade-offs
Instead of: “Rate these features from 1 to 10.”
Ask: “You have 30 minutes a week for managing your finances. Rank these in the order you’d actually spend time on them.”
3. Check stated preferences against actions
Instead of stopping at “Budgeting is important to me”, follow up with: “What budgeting method did you last try? When did you last use it?”
4. Test what people would give up, not just what they’d gain
Instead of: “Would you like better financial planning?”
Ask: “What would you stop doing to make time for better financial planning?”
5. Use the Sean Ellis test for existing products
Ask: “How disappointed would you be if this product disappeared tomorrow?” If 40% or more say “very disappointed”, people are genuinely attached to it.
What good validation surveys actually measure
Good validation surveys look for evidence of what customers already do, not what they say they might do.
What people are already trying to get done: “When you last needed to [specific outcome], what steps did you take?” shows existing habits and workarounds.
How much the problem really hurts: “What’s the most you’ve spent trying to solve [specific problem]?” shows what the pain is worth in real money.
What’s not working today: “What’s frustrating about how you currently [achieve outcome]?” points to specific gaps you could fill.
Whether words match actions: compare what people say they value with what they’ve actually done, to spot the gap between aspiration and reality.
What gets in the way: “What would stop you from [desired behaviour], even if you wanted to?” reveals real-world barriers.
For an even clearer picture of what people do unprompted, look at what they already say in forums and reviews through Community Insight.
When validation surveys work best
Surveys work best as early signals, not as proof. They’re most useful in four situations.
1. Early assumption testing
A quick check on whether a problem is real before you invest in prototypes or deeper research. Mental Mapping is a useful next step for finding out which problems matter most.
2. Prioritising with existing users
Understanding what matters most to customers who already use your product.
3. Choosing between specific options
Helping customers choose between concrete options rather than rating abstract ideas. A co-design workshop does this in more depth.
4. Tracking sentiment over time
Measure changes in satisfaction, likelihood to recommend, or disappointment scores by using the same questions each time.
When you need evidence that people will actually buy, move on to tests that ask for real commitment, such as a letter of intent or an Advance Purchase.
Where should you start?
Take a current project and write two surveys: one that asks what people want, and one that asks what they actually do.
Compare “Would you use [your product]?” with “When you last faced [specific problem], what did you try?” The gap between the two shows which assumptions you need to test.
Surveys work when they test real behaviour and real constraints, not hypothetical preferences. Most teams survey what customers say. Successful teams survey what customers do.
Common questions about validation surveys
Are surveys useful for product validation?
Yes, as an early signal. They’re quick and cheap for checking whether a problem is real or comparing specific options. But survey answers aren’t proof of demand, so follow them up with tests of real behaviour. We’ve seen the same gap between stated interest and action with email metrics.
What is the Sean Ellis test?
It’s a single question for existing users: “How disappointed would you be if this product disappeared tomorrow?” If 40% or more answer “very disappointed”, it’s a strong sign people genuinely value the product.
How do you avoid leading questions in a survey?
Ask about past behaviour rather than future intentions. Don’t mention your product or idea in the question. Give balanced answer options so people aren’t nudged toward the answer you’re hoping for.
Is your business case built on survey results?
If your evidence for a new idea leans on survey responses, it’s worth a conversation about how to test what customers actually do. Miles, Managing Partner at Future Foundry, can walk through whether a validation survey, or another business experimentation method, fits your situation.
Book a call with Miles here.
A validation survey is one method within business experimentation, the discipline of testing what customers actually do before committing budget to a solution. Explore the Validation Survey page in the Toolbox and 50+ other methods in our Experiment Toolbox.
