Varan Join the beta →

Benchmark: schema inference accuracy

Every number here is reproducible. The inputs are public and checksummed, the ground truth comes from somewhere other than us, and the caveats are on this page rather than omitted from it.

The headline

25,000 real GitHub events from the public event stream — a genuinely polymorphic collection, where push, pull-request and issue events share almost no payload structure. Ground truth is every document read, which is definitive by construction. 909 distinct field paths, 346 of them present in under 1% of documents.

“Sampling” is the first 500 documents with no server-side scans — what most tools do, and what Varan itself did before this work. Both arms run over identical data.

Review queue on that collection: 17 fields out of 909 needed a human decision.

Schema inference vs published metadata — 41 datasets

OpenML publishes, for every dataset, each feature's declared type, its exact category set and its missing-value count. We strip all of it, load the rows into MongoDB as bare documents, and measure what is recovered. That is ground truth we did not author.

• Field recall 100% · type accuracy 100%
• Category recall 99.1% — exact value set 98.3%
• Optional-vs-required 99.6%

Then the same datasets are run dirty: 3% of numeric values rewritten as text, the way a collection drifts across application versions. That run does not ask “did you guess right” — a type conflict has no single right answer. It asks whether the conflict was surfaced or silently resolved:

Is the confidence meaningful?

An accuracy figure alone is not that useful. What matters is whether the system knows which of its answers to doubt — because that is what makes a short review queue trustworthy. Accuracy, bucketed by the confidence claimed:

NoSQL → SQL join fidelity

36,273 Wikipedia film documents, with nested cast and genre arrays, joined to the MovieLens movies.csv on title and year through the real pipeline — then scored against a reference join computed straight from the raw files by separate code.

Data sources

All public, no credentials, every file checksummed:

• OpenML — 41 datasets with published feature metadata, from 150 to 48,000 rows, all-numeric to heavily categorical
• GitHub Archive — one hour of the public event stream
• npm registry — real package documents, no two the same shape
• wikipedia-movie-data and MovieLens (GroupLens) — the join corpus

Caveats

These are on the page for a reason. A benchmark without them is marketing.

• The OpenML type axis is somewhat favourable by construction, because numeric values are written to MongoDB as numbers. The category, optionality, calibration, dirty-data and messy-corpus axes are not.
• Exact scans descend six levels and 60 paths per level. Deeper or wider documents fall back to sampling — and say so in the interface.
• 62 false categories across the 41 datasets: low-cardinality columns proposed as enums when they are not. These are cosmetic, but the precision is honest room for improvement.
• The messy-corpus arm computes the server-side aggregations in-process rather than over the wire. They are the same quantities MongoDB returns, handed to the same code, but it is a reimplementation. The OpenML and join arms run against a live mongod.
• Largest corpus is 48,000 rows / 36,000 documents. This has not yet been measured at tens of millions.

What we are not claiming

Not that the impedance mismatch is solved. Discovery — what fields exist, what types they hold, how often — is exact. Interpretation is not, and cannot be: when a field is an integer in most documents and a string in a few, the information needed to decide is not in the data. That residue is real, it is measured, and on the GitHub corpus it came to 17 fields out of 909.

The honest summary is narrower and more useful than “solved”: the guessing has been removed from discovery, and what is left is asked once.

Try it on your own data →