How Enterprises Should Measure the ROI of AI Automation

Claims about AI return are unusually difficult to verify, and the reason is rarely dishonesty. It is that almost nobody measured the process before changing it.

Take the baseline first

Without a pre-deployment measurement there is no denominator. Every subsequent claim rests on a remembered version of how things used to be, and memory is generous about the past when a project is being defended.

A usable baseline captures, for a defined period before any change: volume of cases, cycle time from arrival to resolution, error or rework rate, cost per case including the fully loaded labour component, and the distribution rather than just the average. The distribution matters because automation typically compresses the tail — the slowest cases improve most — and an average obscures that.

This takes weeks and is the single highest-value activity in the programme. It is also the one most often skipped, because it produces no demonstration.

Measure operational outcomes, not model metrics

Precision, recall and evaluation scores are diagnostics. They tell you whether the model is functioning; they say nothing about whether the business is better off.

The outcome measures are the ones from the baseline: cycle time, error rate, cost per case, throughput per person. A system can improve accuracy and worsen cycle time, if the review step it introduced is slower than the work it replaced. That happens more often than it is reported.

Count the costs that get left out

Inference cost is metered and visible. Four others are routinely omitted.

Review labour. If a person checks the output, that time is part of the cost of the automated process, not an overhead.

Exception handling. Cases the system escalates often arrive harder than before, because the straightforward ones were absorbed. Average handling time for the residual queue usually rises.

Maintenance. Model changes, prompt revisions, integration repairs and evaluation upkeep are recurring, not one-off.

Incident cost. The occasional error that reaches a customer carries a real cost, and a deployment of any size will have some.

Excluding these produces the pattern where a system shows strong return in month three and none in month eighteen, without anything having changed except accounting honesty.

Attribute the change properly

Deployments are rarely the only thing happening. Volume shifts, staff change, processes get cleaned up in preparation — that last one frequently delivers a share of the improvement later attributed to the software.

A staged rollout with a holdout group is the practical remedy. Run the new system on part of the volume and the existing process on the rest, over the same period, with the same intake. It costs a little in speed and converts the result from a claim into evidence.

Be honest about what is hard to quantify

Some genuine benefits resist measurement: better consistency, lower key-person risk, work people no longer dread. These are real and should be stated as qualitative findings rather than converted into invented numbers, which is how ROI models lose credibility with finance.

A reporting format that survives scrutiny

Report the baseline period and method; the outcome measures with before and after, including distribution; total cost of ownership with the four components above itemised; the attribution method used; and the qualitative effects, separately labelled.

A report in that shape can be challenged, which is the property that makes it worth reading. The related discussion of build versus buy covers the five-year cost view that belongs alongside it.

References

1. National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1.
2. Infocomm Media Development Authority & Personal Data Protection Commission, Singapore (2020). Model Artificial Intelligence Governance Framework, second edition.