Tools That Claim to Measure AI Impact: What Each Can Actually See

A product category formed in 2026 around a single question: what did the AI coding investment return. The demand is real — boards are asking, and existing platforms cannot answer.

The vendors answering it fall into four architectures, and the architecture determines what any given tool can honestly establish. This is not a ranking; it's a map of what each type can see.

Based on vendor documentation and public sources. Not a hands-on pilot.

The four approaches

Established platforms with an AI module

Jellyfish, LinearB, Swarmia and DX added AI-related reporting to existing products. They operate on pipeline metadata — pull request cycle times, commit volumes, review latency — plus, in DX's case, survey items on AI usage.

Can see: delivery metrics moving after a rollout. Instability, cycle time, rework rate. Perceived reliance, where surveys are used.

Cannot see: which changes came from an assistant. Metadata platforms don't read the diff, which is architectural rather than a gap in the roadmap. Any code-level claim from this category was inferred.

Honest use: correlation. "Instability rose 12% in the two quarters after rollout" is defensible from this data. "AI caused 12% more failures" is not.

Code-level attribution products

Newer entrants reading the contents of changes and attributing authorship at commit or line granularity, sometimes across multiple assistants, sometimes storing provenance in your repository rather than a vendor cloud.

Can see: which changes are AI-authored, and how those changes behave downstream — failure rates, rework, survival.

Cannot see: anything before instrumentation started. Provenance has to be captured when code is written; these tools cannot look backwards. They also don't solve the counterfactual — what would have happened without the tools remains unknown.

Honest use: forward-looking attribution, if you can pass a source access review and have a decision that genuinely needs provenance.

IDE and vendor telemetry

Data from the assistants themselves: suggestions offered, suggestions accepted, characters accepted.

Can see: adoption and engagement, per instrumented tool.

Cannot see: survival. Accepted code gets edited and deleted at rates that vary widely by team, so acceptance overstates output. Coverage is also per-vendor — a team using one assistant in the IDE and another in the terminal reports one and misses the other.

Honest use: adoption reporting. Presenting acceptance rate as productivity is the most common overclaim in this space.

Survey instruments

Asking developers what proportion of their work involved assistance, and how it changed their experience.

Can see: perceived reliance, trust in the code, review burden, whether people feel they understand what they're merging.

Cannot see: a defensible percentage. People estimate their own workflow poorly, and answers shift with framing.

Honest use: the experience dimension, which is where the most interesting signal currently sits. The gap between "we ship faster" and "I trust this code less" is a finding no system data will give you.

What the evidence base looks like

Worth knowing before you evaluate anyone's claims, because the published numbers are what vendors will quote at you.

DORA's 2025 research, drawing on nearly 5,000 responses, finds AI adoption correlating with rising delivery instability alongside improving individual effectiveness, and frames AI as an amplifier of existing organisational capability. Analysis of the 2024 dataset attaches figures: a 25% increase in adoption tracking with a 1.5% drop in throughput and a 7.2% drop in stability.

LinearB's 2026 benchmarks — 8.1 million pull requests across 4,800 teams — found technical debt rising 30–41% after AI tool adoption. A 2026 study of 302,579 verified AI-authored commits across 6,299 repositories found more than 15% introduced at least one detectable issue, with 22.7% still present at the latest revision.

All three are observational. None establishes causation, and none includes a matched baseline for human-authored commits in the same repositories over the same period. A vendor citing them as proof of anything is overreaching, including when the direction happens to be right.

How to evaluate a claim

Four questions that sort the category quickly.

Does it read the code? If not, it cannot attribute. Everything downstream of that is inference.

When does measurement start? Provenance approaches work forward only. Ask directly whether historical attribution is estimated.

Does it cover every assistant your team uses? Per-vendor telemetry undercounts in multi-tool environments by design.

What's the false positive rate? Heuristic detection is common and its accuracy is generally not published. A vendor without an answer is telling you something.

What most teams should do

Probably not buy anything yet.

The question a board is asking — did this pay off — is not answerable at organisation scale with current tooling, and buying a product does not change that. What is answerable is narrower: did a specific workflow improve for a specific team, at what cost, with what effect on review burden. That needs a controlled comparison over a quarter or two and data you already have.

Buy attribution tooling when you have a funded decision that requires provenance and can accept that the number starts from the day you switch it on. Until then, the honest measurement programme is delivery metrics re-baselined at rollout, review time per change, rework rate, and a short survey. Cost: near zero.

Frequently asked

Is there a standard for AI code provenance? Nothing settled. Some approaches store attribution alongside the code in the repository rather than in a vendor database, which is portable in principle. Adoption is early.

Can we estimate retroactively? Only heuristically, with accuracy that is generally not established. Treat any retroactive percentage as directional.

Do the big platforms plan to add code-level attribution? It's an architectural change rather than a feature, and it changes their security posture — which several of them currently market as an advantage. Assume the split persists.

What's the minimum credible measurement? Cost stated precisely, adoption stated honestly, one controlled comparison, and a written log of everything else that changed in the window.