Testing manually before you automate means delivering a new service by hand to real customers before you build the technology to run it. Customers know there’s a person behind it. You learn what they actually ask for, what it really takes to deliver and which parts are worth automating.
We call it a Service Simulation test. In start-up circles, it’s often called a concierge MVP.
Most teams do the opposite. They rely on surveys, focus groups and interviews. They ask “Would you use this?” and “How much would you pay?”, then build what people say they want.
But there’s a big gap between what customers say in a survey or interview and what they actually do in the real world.
What is a Service Simulation test?
You design a customer experience and deliver it manually. Unlike a Behind-the-Scenes Simulation, where the manual work is hidden behind what looks like an automated service, customers know they’re dealing with people.
It’s not about tricking anyone. It’s about understanding what it really takes to solve a customer’s problem before you automate anything.
How a hotel group saved £5.8M by testing manually first
A major hotel chain we work with wanted to build an AI-powered concierge. The plan included personalised recommendations, automated booking help, round-the-clock problem-solving, and tailored experiences during each guest’s stay.
The tech team estimated £6M and 18 months of engineering. Leadership was ready to approve the full build because guest surveys asked for “more personalised service”. We’ve written before about the polite lies customers tell in surveys.
Instead of building the AI version first, we encouraged them to set up a simple chat service where real team members answered guest requests by hand. They tracked every interaction, including response times, how often problems were solved, guest satisfaction, and what each request type needed.
What the data showed:
70% of guest requests were for basic information, like Wi-Fi passwords and directions
Only 15% actually needed “intelligent” recommendations
Guests valued speed over sophistication
Many requests that looked like a job for AI still needed human judgement
What they built instead
After six weeks, the team understood what genuinely needed automating and what worked better with people involved. They built a simple chatbot for basic information and invested in extra training for staff handling more complex requests.
The total cost was £200K, and it gave guests a much better experience than the original £6M AI plan would have.
Planning to automate a service or build an AI tool?
Before you commit to the build, it’s worth a conversation about what customers actually need and how to test it cheaply first. Miles, Managing Partner at Future Foundry, can walk through whether a Service Simulation test, or another business experimentation method, fits your situation.
Book a call with Miles here.
What a manual test tells you that nothing else can
Surveys and interviews tell you what customers might do. A manual test shows you what they actually do when they need help.
1. What customers really ask for
You see real behaviour when customers need something, not what they predicted in a survey. In the hotel’s case, that meant discovering most requests were simple.
2. What it takes to deliver
You learn first-hand how much time, skill and effort each request takes, and whether you can deliver it profitably.
3. What’s worth automating
You get hard data on what to build, what to automate and what to keep human, before you spend anything on technology.
How to run a Service Simulation test
1. Start with one specific customer problem
Not “guests need better service” but “guests wait 15 minutes for basic information like Wi-Fi passwords”. The more specific and painful the problem, the clearer your measure of success.
2. Design the manual process
Map out step by step how you’d solve that problem by hand. What information do you need? How long does each step take? What could go wrong?
3. Set clear success measures
Decide what good looks like: customer satisfaction, time to resolve, willingness to pay. Without clear measures, you can’t prove you’ve solved anything.
4. Start small and be open about it
Test with five to ten customers who know they’re getting a manual service. Focus on learning, not impressing. You’re proving the value exists before you automate it.
5. Document everything
Track every step, every delay and every customer reaction. That record is what tells you what to build, what to automate and what to keep manual.
Where should you start?
Pick a service project you’re working on. Instead of building the automated version, work out how you’d deliver the same value by hand for five to ten customers. Focus on one specific use, document every step and charge real money if you can. An Advance Purchase is one way to test whether people will pay.
You’ll find out whether customers actually value what you’re planning to build, and what it really takes to deliver it profitably.
Common questions about testing manually before you automate
What’s the difference between a Service Simulation and a Behind-the-Scenes test?
In a Service Simulation, customers know a person is delivering the service. In a Behind-the-Scenes Simulation, also called a “Wizard of Oz” test, the service looks automated, but people do the work behind the scenes. Use a Service Simulation when you want to learn openly what customers need. Use Behind-the-Scenes when you want to see how people react to what feels like the finished product.
Is manual testing useful for AI projects?
Yes, it’s one of the best ways to test an AI idea before you build it. Having people handle requests by hand shows you which tasks are common, which are simple and which genuinely need intelligence. As the hotel group found, the answer is often much less AI than you expected.
How long should a manual service test run?
Run it long enough to see the full range of what customers ask for. The hotel group ran theirs for six weeks. Start small, and stop once new requests stop teaching you anything new.
Planning to automate a service or build an AI tool?
Before you commit to the build, it’s worth a conversation about what customers actually need and how to test it cheaply first. Miles, Managing Partner at Future Foundry, can walk through whether a Service Simulation test, or another business experimentation method, fits your situation.
Book a call with Miles here.
A Service Simulation test is one method within business experimentation, the discipline of testing what customers actually do before committing budget to a solution. Explore the Service Simulation test page in the Toolbox and 50+ other methods in our Experiment Toolbox.



