Metadata or Code-Level Analytics: Which Architecture Do You Need?

Every engineering analytics platform sits on one side of a line. Some read metadata — pull request cycle times, commit volumes, review latency, issue tracker fields. Others read the code itself.

That single architectural choice determines what questions the tool can answer, how long your security review takes, and how much of the vendor's pitch is relevant to you. It's also the axis vendors are least honest about, because each side describes its own constraint as a feature.

What's the actual difference?

Metadata platforms — the established engineering intelligence category, including Jellyfish, LinearB, Swarmia and DX — connect to your repository and issue tracker APIs and read the record around changes. How large the diff was, how long review took, when it merged, what it was tagged as. They do not read what the diff contains.

Code-level platforms read the contents of changes. This lets them attribute authorship at line or commit granularity, assess characteristics of the code itself, and connect specific changes to downstream outcomes.

The consequence is straightforward. Metadata answers questions about flow. Code-level answers questions about content. Most questions engineering leaders actually have are flow questions.

What can each one answer?

Note the last row. Neither architecture gets you developer experience data. That comes from surveys, which is why platforms in both camps have added a survey product, and why DX built its position around one.

Note also the "if instrumented" qualifier. Code-level attribution generally cannot look backwards — provenance has to be captured when the code is written. Buying a code-level platform today does not give you an answer about last year.

How do the vendors frame it?

Predictably, and in opposite directions.

Metadata vendors present not reading your code as a security advantage: no source access, smaller attack surface, easier procurement. Code-level vendors present the same fact as a capability ceiling: metadata physically cannot see what the pipeline moved.

Both statements are true and both are marketing. The security argument is genuinely material at organisations where source access triggers a lengthy review — and at many enterprises it does, adding weeks to procurement. The capability argument is genuinely material if you have a question that requires provenance. Neither one settles the decision on its own.

What should settle it is a question you can name. If you cannot articulate a decision that depends on knowing what's inside the diff, you're buying a capability you won't use and paying for it in procurement time.

Which should you buy?

Buy metadata if: your questions are about delivery flow, bottlenecks, allocation and trends. Your security review is strict. You have fewer than a few hundred engineers. You want a fast implementation. This covers the large majority of teams.

Buy code-level if: you have a specific, funded question about code provenance — usually AI ROI reporting to a board — and you can pass a source access review. You're prepared for the number to be forward-looking only.

Buy both, or neither, if: you're at the "we should measure something" stage. In that case a period with free tooling and a manual read of your own data will tell you what question you actually have, which is worth more than either purchase.

Two specialised tools often beat one platform. A process tool for pipeline flow plus a survey tool for experience covers more ground than one product claiming to do everything, and the pieces can be replaced independently.

The buying mistake to avoid

Most buyers discover three months into a contract that the platform answers a question they didn't ask.

The way to avoid it is to write down the question before the demo, in a form specific enough to be wrong. Not "is our team productive" — that's unanswerable. Something like: "why did this project take three times the estimate", "which services are slowing product delivery", or "where is our review capacity going".

Then ask each vendor to show you that question answered on your data, not on their demo instance. Vendors that can't will say the pilot is too short. That answer is itself informative.

Frequently asked

Do metadata platforms give any AI signal? They can show delivery metrics moving after a rollout — instability, cycle time, rework. That's real and useful. It isn't attribution, and a vendor presenting it as AI ROI measurement is overreaching.

Is code access actually a security risk? It's a security review, which is different. The risk depends on the vendor's architecture — some keep attribution data in your repository rather than their cloud, which changes the conversation materially. Ask where the data lives, not just whether it's encrypted.

Can I switch later? Metadata to code-level, yes — you keep your historical delivery metrics and start provenance from the switch date. Code-level backwards is trivially easy since you're giving up capability, not data.

How long should a pilot be? Thirty days per platform is the commonly used window and it's about right — long enough to see real cycles, short enough to run several. Running pilots sequentially on the same team gives you a cleaner comparison than running them in parallel on different teams.