Measuring what an AI agent actually saves, six months in

The figure quoted before an automation project and the figure observed six months later are never the same.

That is not necessarily a failure, and it is almost never dishonesty. It is that the original calculation was not counting the same things reality does.

Here is how to measure without flattering yourself, in either direction.

The baseline is taken before, or it does not exist

This is the rule that decides everything else, and it is the one that gets skipped.

A measurement taken after go-live compares the present against a memory. Memory is unkind to the past: it recalls the days the task was painful, not the average. The apparent gain is therefore overstated, sometimes by a factor of two, and nobody can contest it because there is nothing to compare against.

On the engagements we run, taking that baseline is the first deliverable, before anything is built, precisely because it cannot be reconstructed afterwards and because a project whose current cost nobody can state is a project nobody has justified.

The baseline does not need to be sophisticated. Two weeks, a sample of real cases, three columns: how many times the task was done, how long each took, how often it had to be redone.

That last column is the one people omit and it is frequently the most interesting. A task that takes ten minutes but has to be redone one time in five does not cost ten minutes.

The three items in an honest calculation

Processing time itself. The obvious item, and the one that appears in every vendor promise.

Review time. An agent produces something a person reads, at least at first and usually forever. That is legitimate and it does not disappear. An agent that drafts a reply in ten seconds, reviewed and corrected in two minutes, has not eliminated an eight-minute task: it has reduced it to two. The gain is six minutes, not eight.

Time on unhandled cases. No agent covers every situation. The cases it refuses or handles badly return to a human, and they often return in a worse state: somebody has to understand what the agent did before they can take over. This overhead is real and it falls precisely on the most complex cases, which are the longest ones.

An honest calculation subtracts the last two from the first. It produces a smaller number that survives questioning, which is worth more than a flattering one that collapses at the first precise question.

The two gains nobody counts, and they are real

Symmetry is required here: if you add the hidden costs, you must add the hidden benefits.

Latency, as distinct from effort. A request handled continuously by an agent gets an answer in the evening and on the weekend. The human time saved may be small, but the delay may go from two days to ten minutes. On a commercial response, that is not time saved, it is revenue, and it appears in no productivity table.

Consistency. An automated process does the same thing every time. A human process varies by person, by time of day and by workload. On tasks where variation is expensive, consistency is often worth more than speed, and it is easier to demonstrate than to price.

Neither converts cleanly into money. Better to name them and leave them outside the numeric total than to convert them with an invented coefficient: an assessment that mixes measurements with estimates loses the credibility of both.

Why six months, and not six weeks

The early weeks lie in both directions.

They overstate, because novelty drives usage beyond what is useful, including where the tool adds nothing.

They understate, because every unanticipated case arrives at once and each one demands attention.

By six months usage has settled. People have found where the tool helps and where it does not, the workarounds have formed, and what you measure then is what the following year will look like.

Those workarounds deserve particular attention: they are the most useful information in the whole assessment. When a team has started skipping the tool in certain cases, they have identified for free the limits the analysis missed. Ask which cases, and why, in a way that does not sound like an accusation.

Adoption explains most disappointments

Before concluding that an agent has underdelivered, check whether it is being used.

A tool used on three quarters of the intended cases delivers three quarters of the expected gain, which is enough to turn a good result into a disappointing one without a single technical assumption having been wrong.

And where adoption is low, the cause is nearly always the same: the tool requires stepping outside the normal flow of work. One more interface, one more tab, one more step. An automation that lives inside the software people already use gets adoption for free. One that lives beside it gets adoption that decays. That is the third of the three questions worth asking before building anything.

The check itself needs no instrumentation: the number of real runs against the number of eligible cases is enough to settle it.

What a poor result should trigger

A measured gain well below expectation does not mean stopping. It means finding where the value went, and there are three places.

In the review, if it is heavier than planned. The fix is to narrow the agent’s scope to the cases where it is reliable, and accept that the rest does not go through it. An agent handling sixty percent of cases without review beats one handling everything that is read in full.

In the unhandled cases, if they are more numerous than estimated. Either fixable, or a sign that the task was less regular than believed.

In the adjacent step, when the automation moved the bottleneck without removing it. This is the most common and the most misdiagnosed: the time was genuinely saved, and it now waits somewhere else.

The only case that cannot be fixed is a task that was never expensive enough to justify the project. That is knowable in advance, by timing it, which is the first reason to take the baseline.

Further reading

• Why AI pilots stall before production: the three questions asked before building
• Self-hosted model or vendor API: the most discussed line, rarely the heaviest
• Making the case for technical debt: the same discipline, on a different subject
• Acceptance criteria that end the argument: defining the finish line before starting
• AI automation: how we scope one task and measure it