Why DORA Metrics Can't Tell You If AI Is Working

Your DORA dashboard went green after rolling out Copilot, and your engineers say they feel slower. Here's why both can be true.

DORA measures pipeline metadata: how often you deploy, how long changes take, how often they fail, how fast you recover. None of those fields record who — or what — wrote the code.

That gap is why engineering leaders keep landing in the same uncomfortable meeting. The dashboard is green. The board wants a number for the AI investment. And three senior engineers, asked directly, say the work feels worse than it did a year ago. All three of those things can be true at once, and DORA is structurally incapable of resolving the contradiction.

Provenance. A pull request looks identical whether a human wrote it, an assistant drafted it, or an assistant wrote it and a human rewrote half. Attribution requires reading the diff, and DORA-based platforms operate on metadata by design.

Counterfactuals. DORA can show that deployment frequency rose 20% after your rollout. It cannot tell you what would have happened without it. Teams adopt AI tools during the same quarters they hire, reorganise, and pay down debt.

Where the work moved. The most common real effect of coding assistants isn't that work disappears — it's that it relocates. Time saved writing code reappears as time spent reviewing code. Lead time can stay flat while the composition of it changes completely, and DORA reports one number for both.

Cost. Nothing in the framework touches licence spend, token spend, or the hours spent on prompt scaffolding and cleanup.

Isn't a rising change failure rate a clear signal?

It's a signal. It isn't the signal most people read it as.

DORA's own 2025 research found that increased AI adoption correlates with increased delivery instability, even while individual effectiveness improves. Analysis of the 2024 dataset attached figures: a 25% increase in AI adoption tracked with a 1.5% decrease in throughput and a 7.2% decrease in stability.

Read carelessly, that says AI makes teams worse. Read properly, it says something more useful. The 2025 report — drawing on nearly 5,000 survey responses and over 100 hours of qualitative work — frames AI as an amplifier rather than a solution. It accelerates teams with solid foundations and magnifies the dysfunction of teams without them.

The mechanism is usually downstream capacity. Code arrives faster; review, testing and QA were sized for the old arrival rate; defects escape. Your change failure rate is reporting a bottleneck in your review process, not a verdict on the tooling.

This matters commercially. If you take "change failure rate rose 7%" to a budget conversation as evidence the AI investment failed, you will likely cut the wrong thing.

Why did DORA itself change?

Because the framework's authors saw the same problem.

In 2025 DORA stopped expanding its own acronym and renamed the annual report from Accelerate State of DevOps to the State of AI-assisted Software Development. It replaced the familiar four-tier performance ranking with seven team archetypes, on the reasoning that a single ladder no longer described what teams were actually doing. It also published an AI Capabilities Model — seven foundational practices spanning technical, cultural and process dimensions — as a companion to the metrics.

The signal in all of that is worth reading directly: the research programme that defined delivery measurement responded to AI by adding a diagnostic model alongside the metrics, rather than by adding an AI metric. That is an admission that the metrics alone don't answer the question.

What actually answers it?

Nothing single. A workable programme layers three things.

Delivery metrics, re-baselined. Keep DORA, but reset your baseline at the rollout date. Comparing across a workflow change that fundamental produces confident nonsense. Watch the throughput and instability dimensions separately rather than as one health score — they are moving in opposite directions, and averaging them hides the whole story.

Developer experience data. Structured surveys covering focus time, CI/CD friction, review burden and perceived quality. This is where the "feels worse" signal lives, and it's the one that predicts attrition. If your engineers report spending more time reviewing code they don't trust, that's an operational finding, not a mood.

Code-level attribution, if the question is ROI. If a board wants the return on AI spend, metadata cannot get you there. That requires tooling that reads the diff and attributes changes to specific assistants — a different and newer category of product, with its own accuracy caveats.

Scope the claim honestly. Most teams don't need a global ROI figure. They need to answer whether a specific workflow got better for a specific team. That question is answerable with a controlled comparison over one or two quarters, and it survives scrutiny in a way that an org-wide percentage does not.

Frequently asked

Should we stop tracking DORA after adopting AI tools? No. DORA still does what it always did — shows where work gets stuck. The mistake is treating it as an AI scorecard. Keep it, re-baseline it, and stop reading it as a single health number.

Our DORA metrics improved after AI rollout. Isn't that proof it worked? It's consistent with it working. It's also consistent with a hiring wave, a completed migration, or a quieter quarter. Without a controlled comparison, an improvement is a correlation with several plausible causes.

Can any platform measure AI impact from metadata alone? Not at the code level. Platforms operating on pull request and commit metadata can show correlated movement in delivery metrics. Determining which lines came from an assistant requires reading the diff, which is a different architecture.

What's the minimum viable measurement setup? Delivery metrics you already have, plus a quarterly developer experience survey, plus a written note of what changed and when. The third one costs nothing and prevents most of the misattribution.