blinkx analytics

AI measurement

What to Tell the Board About AI ROI

You've been asked for a return figure on AI coding tools. Here's a defensible answer, why the confident version is worse than the honest one, and what to bring instead.

Someone has asked you what the AI coding tools returned. You have licence costs, a delivery dashboard, and engineers with strong and contradictory opinions.

The temptation is to produce a percentage. Don't. Here's what to bring instead, and why the honest version is the stronger position in the room.

Why the confident number is the worse answer

Because it won't survive the first competent follow-up question, and you'll have spent your credibility on a figure you can't defend.

Three problems make a clean ROI figure unavailable with normal tooling.

Attribution. Nothing in git records whether a line was typed, suggested, generated, or generated and then rewritten. Metadata platforms — which is most of the category — cannot see inside the diff. A number that claims code-level attribution without code-level instrumentation was estimated.

Counterfactual. You can show throughput rose. You cannot show what would have happened without the tools. Teams adopt AI during the same quarters they hire, reorganise, and finish migrations.

Displacement. The most common real effect isn't work disappearing but relocating — time saved writing code reappearing as time spent reviewing it. A single lead time figure reports both and distinguishes neither.

If you present a percentage and a board member asks how you controlled for the hiring wave, the meeting stops being about AI.

What the external evidence actually shows

Bring this, because it reframes the question from "did it work" to "what does it cost to make it work."

DORA's 2025 research, drawing on nearly 5,000 responses, finds AI adoption correlating with increased delivery instability alongside improved individual effectiveness. Its central framing is that AI is an amplifier rather than a solution — accelerating organisations with solid foundations, magnifying dysfunction in those without.

Analysis of the 2024 dataset puts numbers on the trade: a 25% increase in AI adoption tracked with a 1.5% decrease in throughput and a 7.2% decrease in stability.

On maintenance burden, LinearB's 2026 benchmarks — 8.1 million pull requests across 4,800 teams — found technical debt rising 30–41% following AI tool adoption. A separate 2026 study of 302,579 verified AI-authored commits across 6,299 repositories found more than 15% introduced at least one detectable issue, with 22.7% of those still present at the latest revision.

The honest reading, and the one worth presenting: these are observational findings, not causal ones, and the mechanism most consistent with them is capacity mismatch. Code arrives faster; review, testing and QA were sized for the previous rate. That points at an investment, not a reversal.

What to actually present

Cost, precisely. Licences, token spend, and the engineering time spent on setup, prompt scaffolding and cleanup. This is the one number you can state without qualification, and stating it precisely buys credibility for everything that follows.

One controlled comparison. Not an org-wide figure. One team, one workflow, one or two quarters, with a stated baseline and a written note of everything else that changed in the window. A narrow claim you can defend beats a broad one you can't.

Adoption, honestly. How many engineers use the tools weekly, and for what. Acceptance rates from IDE telemetry measure engagement, not output — say so rather than letting the number imply more than it supports.

The instability picture, with the mechanism. If your change failure rate moved, present it alongside review time per change. If total review hours held steady while changes per week rose 30%, review depth per change fell by roughly the same proportion. That's your bottleneck, computed from data you already have, and it turns a worrying chart into a fundable proposal.

What you'd need to answer the question properly. Provenance instrumentation, and the fact that it only works forward. If the board wants a defensible attribution number in twelve months, the decision to instrument has to happen now.

How to frame it

The strongest available position is not "it worked" or "it didn't." It's this:

We can state cost exactly. We can state adoption exactly. On output we have a controlled comparison for one team, which showed X, with these caveats. Industry research indicates delivery instability rises alongside effectiveness, and the mechanism is downstream capacity rather than code quality — which means the return depends on an investment we haven't yet made. Here's what that investment is, and here's what instrumentation we'd need to answer this question properly next year.

That answer survives scrutiny, identifies a fixable constraint, and asks for something specific. A percentage does none of those things.

The trap to avoid

If you present "change failure rate rose 7%" as evidence the AI investment failed, the likely outcome is cutting the tools — while the actual constraint sits in review and testing capacity, which then goes unfunded.

The instability is real. The conclusion most people draw from it is wrong. Bringing the mechanism alongside the number is what stops the wrong decision from being made in your meeting.

Frequently asked

Can any tool give a real ROI number? Code-level provenance tooling can attribute changes to specific assistants, which gets you closer than metadata. It still doesn't solve the counterfactual, and it only works from the day it's turned on.

What if the board insists on a percentage? Give a range with the assumptions stated, and say plainly what would make it precise. A stated range with visible assumptions is defensible; a point estimate without them is not.

Isn't this too hedged for an executive audience? Executives are used to evaluating investments under uncertainty — it's most of the job. What they don't tolerate is a confident number that collapses under a follow-up question.

How long before a real answer is possible? One to two quarters for a controlled comparison on one team. Longer for organisation-wide attribution, and only if instrumentation starts now.