Why AI pilots stall before production, and the three questions that predict it

The pattern is consistent enough to be predictable. A pilot is run, it works, everybody is pleased, and six months later nothing is in production.

The post-mortem usually blames the technology. In the work we see, that is almost never where the problem was.

What a pilot proves, and what it does not

A pilot demonstrates that a model can perform a task on examples somebody selected. That is a real result and it is worth having.

A production system has to do three other things, none of which the pilot tested.

Handle the cases nobody selected. The pilot ran on clean examples, because clean examples were available and demonstrating on messy ones is unrewarding. Real work is mostly the messy ones.

Fit inside a process people already follow. The pilot lived in its own interface, which nobody minded because using it was the point of the exercise. In production, the same interface is one more tab, and one more tab is where adoption goes to die.

Be answerable when it is wrong. In a pilot, a wrong answer is a data point. In production, a wrong answer reaches a customer, and somebody has to own that.

A pilot that skipped all three is not a smaller version of the system. It is a different exercise that happens to use the same model.

The three questions that predict the outcome

Asked before anything is built, these three predict production better than any demonstration.

One. What share of cases must the tool handle for this to be worth doing, and what happens to the rest?

The second half is the one that gets skipped. Every automation leaves a remainder, and that remainder comes back to a human, often in a worse state than if the tool had never touched it: somebody now has to understand what the tool did before they can pick up the work.

If the plan is to look at the exceptions later, the project will stall at exactly that point, because the exceptions are where the difficulty was concentrated all along.

Two. Who checks the output, and is their time counted?

An agent that produces an answer in ten seconds, reviewed and corrected in two minutes, has not removed an eight-minute task: it has reduced it to two. That is a genuine gain of six minutes and it is worth having. But a business case built on eight will not survive its first review, and the person who built it will not be trusted with the second project.

Naming the review cost at the start is what makes the case defensible when somebody eventually measures it, which somebody should, six months in.

Three. Does this live inside the software people already use?

An automation that requires opening something new gets an adoption curve that decays. An automation that appears inside the tool where the work already happens gets adoption without anybody being asked.

This is a plumbing question, it is unglamorous, and it decides more outcomes than model selection does. It is also the reason integration work should be scoped as part of the project rather than treated as a phase two that never arrives.

The two failure modes worth naming

The pilot that was never going to scale. It ran on a sample somebody prepared, with a person watching every output, in a context where errors had no consequence. Everything that made it succeed is absent in production. The tell is that nobody can say what the tool does when the input is not what it expected.

The pilot with no owner for the process. The technology function built it, it works, and nobody on the business side can decide that the work is now done differently. An automation is a change to how work happens, and that change needs somebody with the authority to make it. Where that person does not exist at the start, the rollout stalls politely and indefinitely, and no amount of technical quality rescues it.

What a first step should look like instead

One task. One team. Live data, not a prepared sample. And a measurement taken before it starts.

That last item is the one that costs nothing and is almost always skipped. Two weeks of timing the task as it is done today, on real cases: how often, how long, how often it has to be redone. Without that baseline, any later assessment compares the present against a memory, and memory is generous to the new tool.

If nobody can define the measurement, that is a finding rather than an obstacle. A task whose current cost cannot be described is a task nobody has established is worth automating, and discovering that before spending is the cheapest possible outcome.

Where the money actually goes

One correction to a common assumption: on the projects we run, the model is rarely the largest line.

Preparing the data, integrating with the existing process, and verifying the output routinely cost more than the model calls. Which means the choice between a hosted model and a vendor API, which absorbs most of the discussion, is decided on the smallest item in the budget.

That is worth knowing before the first meeting, because a conversation that spends its time on model selection is a conversation that has not yet reached the part that determines whether anything ships.

Further reading

• Measuring what an AI agent actually saves: the assessment six months in
• Self-hosted model or vendor API: the decision that matters less than it seems
• Build or buy: the same discipline, on software generally
• Writing requirements that produce comparable quotes: how to scope the integration properly
• AI automation: how we take one task from pilot to production