Why Your DORA Metrics Are Lying to You

A DORA dashboard always produces numbers. Whether those numbers describe your delivery system or your tool configuration is a separate question, and most teams never check.

Here are the seven ways the numbers go wrong, in rough order of how often they do.

1. Your definition of "deployment" is doing the work

Deployment frequency depends entirely on what counts as a deployment. Every merge to main? Every production release? Every environment promotion? Feature flag flips?

Teams using trunk-based development with flags can show elite deployment frequency while shipping user-visible change monthly. Teams behind a release train look poor while delivering steadily. Neither number is wrong; they're answering different questions.

Check: write down your definition. If nobody can state it without looking it up, the metric is measuring your integration configuration.

2. Lead time starts somewhere arbitrary

Commit to production is the standard definition, and it excludes everything before the first commit — refinement, design, waiting for a decision, sitting in a backlog.

In most organisations that excluded portion is the larger part. A team with excellent lead time and a six-week intake queue is a slow team with a fast pipeline, and DORA reports the pipeline.

Check: measure idea-to-production for five recent pieces of work by hand. Compare it to your dashboard. The gap is what you're not seeing.

3. Change failure rate depends on incident hygiene

Change failure rate requires knowing which deployments caused problems. That comes from your incident data, and incident data reflects reporting discipline as much as reality.

Teams that log every degradation look worse than teams that only log outages. A team improving its change failure rate may simply have stopped filing minor incidents. This metric measures your incident culture and your delivery quality together, and it does not distinguish between them.

Check: did your incident count change at the same time as your failure rate? If yes, you're looking at reporting, not reliability.

4. Recovery time measures your rollback, not your resilience

The metric — now called failed deployment recovery time — captures restoration after a failed deployment, not recovery from incidents generally. The rename was deliberate, because the old MTTR label encouraged teams to measure the wrong thing.

Excellent numbers here often mean you have good automated rollback. That's genuinely valuable. It is not the same as a resilient system, and treating it as such lets real fragility hide behind a good chart.

5. Averages hide the distribution

A team with a two-day median lead time and a p90 of three weeks is not a fast team. It's a team that is fast at easy work and unpredictable at hard work — and the second fact is the one that breaks roadmaps.

Check: look at p75 and p90 alongside the median. If nobody has, you have been reading a summary statistic as if it were a description.

6. The baseline is invalid after any workflow change

Adopting AI coding assistants, changing the review process, splitting a monolith, reorganising teams — each invalidates comparison across the boundary. Comparing quarters that span one produces confident nonsense.

This is acute right now. DORA's own research finds AI adoption correlating with increased delivery instability alongside improved individual effectiveness, and its 2025 report frames AI as an amplifier of whatever the organisation already was. Reading a post-rollout change failure rate against a pre-rollout baseline, without noting the change, produces a conclusion that will not survive scrutiny.

Check: keep a dated log of what changed. It costs nothing and prevents most misattribution.

7. Nobody has looked at the raw data

The most common failure and the least discussed. A platform is connected, dashboards populate, numbers get reported — and no one has ever verified that the underlying events are being classified correctly.

Deployments that aren't deployments. Bot commits counted as work. Repositories excluded by a filter someone set during onboarding. Merged-and-reverted changes counted twice.

Check: take one metric, one team, one month. Compute it by hand from raw data. If it matches, trust the platform. If it doesn't, you have just learned something more valuable than any dashboard.

What good use looks like

The teams getting value from DORA share a few habits.

They treat metrics as questions rather than scores. A number moving prompts an investigation, not a conclusion.

They keep the throughput and instability dimensions separate. Averaging them into one health score hides the tension the framework exists to expose — particularly now that AI adoption is pulling the two apart.

They pair delivery metrics with something qualitative. The dashboard tells you what moved; developers tell you why.

They re-baseline deliberately and write down why.

And they accept that a metric which never changes anyone's decision is a metric to delete. Most dashboards in this category could lose half their tiles and lose nothing.

Frequently asked

Does this mean DORA is useless? No. It's the best-evidenced measurement framework in software delivery. It's also frequently implemented without anyone checking what the numbers describe.

How often should we validate the data? Once at implementation, once after any integration change, and once a year. Manual verification of one metric for one team over one month takes an afternoon.

Our numbers look great. Is that suspicious? Worth checking. Uniformly excellent DORA metrics alongside teams reporting frustration usually means the metrics are measuring a narrow slice of a longer process.

Should we benchmark against industry figures? Sparingly. Published benchmarks aggregate wildly different definitions of the same metrics. Your own trend line over time is a far more reliable comparison than someone else's median.