A better model is not enough.
Data needs a better harness.

When we started building Vuon, frontier models were already writing impressive SQL. That wasn't the problem. The problem was that they kept getting it subtly wrong — not in ways that crashed a query, but in ways that produced a confident, plausible, runnable answer that was incorrect for the specific business we were analyzing.

The failure pattern we kept seeing was this: a query that is completely correct for one company in a vertical is wrong for a different company in the same vertical, because of how that organization defines its metrics. Cohort analysis is a simple example. To avoid survivorship bias you need to count users at the time of their conversion event, not at the current date. But that's not universal — different organizations define conversion differently, track it in different tables, and apply different exclusions. A model without access to those specifics has no reliable way to know which answer is correct. It will pick one, and it will sound confident either way.

We realized the gap wasn't the model. The gap was the absence of corrective infrastructure.

In software engineering, correctness has a harness around it. A compiler tells you when code is type-wrong before it runs. A linter flags structural problems. A test suite catches regressions. The codebase itself provides context that constrains what valid code looks like. These mechanisms don't just help developers write better code — they give agents a tight feedback loop. When a coding agent gets something wrong, the environment tells it.

Data has none of that. You can feed schema documentation and metric definitions into a prompt, and we do — but that turns out to be necessary, not sufficient. It doesn't scale without a system behind it. It can't enforce anything. And it can't tell a model that a query ran successfully but is still wrong for this business, on this question, at this point in time.

That's the layer we've spent the last year building.

The evaluation results confirmed what we were observing in practice: once tasks require business-definition fidelity, correction loops, and reproducibility, the gap between a naive setup and a purpose-built harness is large and consistent.

Each scenario compares Vuon with a Claude baseline that receives the same schema and metric context, but not Vuon's semantic compiler, validation passes, or calculation-versioning layer.

These are internal results, not the final benchmark package. They illustrate the recurring pattern: once tasks require definition fidelity, correction loops, and reproducibility, the baseline falls behind sharply.

A fair benchmark already assumes both systems can execute queries and access the same domain context. The harder question is whether the system can tell the agent its SQL is wrong even when the database runs it, or that a narrative is unsupported even when the numbers look superficially plausible.

We use the term SQL compiler intentionally. It is not a thin query transport. It is a context-aware compiler that can tell the agent when candidate SQL is out of alignment with metric definitions, segment semantics, or exclusion policy.

Post-execution validation matters just as much. Some queries are syntactically correct and semantically plausible, yet still wrong because they produce impossible transitions, unstable denominators, or implausible distributions.

Reproducibility is part of the correctness loop. Tracked and versioned calculations let the agent rerun parts of its own analysis and catch inconsistencies before delivery.