The Metrics That Stopped Working When AI Started Writing Code

Most engineering metrics are proxies. They don't measure the thing you care about; they measure something that used to correlate with it reliably enough to act on.

AI-assisted development broke several of those correlations. The metrics still produce numbers, which is the problem — a broken proxy doesn't announce itself, it just quietly starts meaning something else.

Broken: code volume in all its forms

Lines of code, commits, characters written. These were always poor measures, but they held a weak relationship to effort. That relationship is now essentially severed: generating a hundred lines and typing a hundred lines are different acts requiring different amounts of thought, and neither the commit nor the diff distinguishes them.

Anything derived from volume — churn ratios, contribution charts, activity heatmaps — inherits the break.

Broken: keystroke and editor-time measurement

Tools measuring keystrokes, time in editor, or coding hours are degrading fastest. Time in the editor no longer tracks work done in either direction: an engineer can produce a substantial change in minutes, or spend an hour reviewing generated code without touching the keyboard much.

If your organisation runs anything in this family, it's now measuring something close to noise, and it will keep producing plausible charts while doing so.

Broken: diffs per engineer as a speed proxy

This one matters because Core 4 uses it as the headline Speed metric, deliberately, on the grounds that it communicates to non-technical stakeholders in a way lead time doesn't.

The framework's authors acknowledged the gaming risk openly and argued the other three dimensions constrain it. That defence was reasonable when producing a diff required proportional effort. It's weaker now: assistants make it cheap to produce more, smaller changes, so the metric can rise substantially with no change in delivered value and no intent to game anything.

Still usable, but only with the counterweights genuinely in place, and only re-baselined after your rollout.

Bent, not broken: cycle time

Lead time and cycle time still measure something real. What changed is the composition.

Work displaced rather than disappeared. Time saved writing code reappears as time spent reviewing it. Cycle time can hold perfectly flat while the internal distribution shifts substantially — and a single number reports both states identically.

What to do: break cycle time into stages and watch the stages, not the total. Coding time falling while review time rises is the signature pattern, and it's invisible in the aggregate.

Bent: change failure rate

Still meaningful, now frequently misread.

DORA's research finds AI adoption correlating with rising delivery instability alongside improving individual effectiveness. Read as a verdict on the tooling, that's misleading. The mechanism most consistent with the evidence is capacity mismatch: code arrives faster, while review, testing and QA were sized for the previous arrival rate.

Your change failure rate is now partly reporting a bottleneck downstream of code generation. Same metric, different meaning, and a decision made on the old interpretation cuts the wrong thing.

Broken: any baseline that spans your rollout

Not a metric, but the most common source of wrong conclusions right now.

Adopting coding assistants is a workflow change on the scale of changing your branching model. Comparing quarters across that boundary produces confident nonsense in whichever direction the noise happens to fall.

What to do: re-baseline at the rollout date and keep a dated log of what changed. It costs nothing and prevents most misattribution.

What still holds up

Review time per change. Total review hours divided by changes merged. If total hours held steady while changes rose 30%, review depth per change fell by roughly the same proportion. This is the single most useful measure right now — it moves early, it points at the mechanism, and it computes from data you already have.

Rework rate. DORA added it in the 2024 report: the share of unplanned deployments made to fix user-visible problems. It's a good early indicator that verification capacity is undersized, which is precisely the failure mode AI adoption produces.

Outcome measures. What shipped, what changed for users, what the business got. These were always the honest measures and they're unaffected, because they sit downstream of how the code was produced.

Developer experience surveys. Perception data measures the lived experience directly rather than inferring it from artefacts. When the artefacts stop meaning what they used to, direct measurement gains relative value.

The general principle

Every proxy metric carries an implicit assumption about the relationship between the thing measured and the thing meant. AI changed several of those assumptions without changing the metrics.

The practical exercise, worth an hour: take each metric on your dashboard and write down what it's a proxy for and why that proxy held. Then ask whether the reason still applies. Most teams find two or three that don't, and deleting them is a net gain.

Frequently asked

Should we stop using diffs per engineer entirely? Not necessarily, but re-baseline it and make sure the counterbalancing dimensions are actually in place rather than nominally present. Alone, it now moves for reasons unrelated to delivered value.

Is any activity metric still useful? As adoption and engagement signals, yes. As output or productivity measures, they were weak before and are weaker now.

How do we know if our metrics broke? The signature is a dashboard that looks stable or improved while the people doing the work report the opposite. That gap is the symptom.

What replaces the broken ones? Stage-level cycle time, review time per change, rework rate, and outcome measures. Fewer metrics, closer to the mechanism.