Every platform ships a metrics dashboard. Ours is nearly empty, and that is a considered decision rather than an oversight.
The one number
Time spent reviewing a specification, divided by time spent reviewing the equivalent code.
Target: below 0.3. Acceptable: below 0.45. Above 0.6 and the approach is not earning its place.
This is the only metric that decides whether the platform works, and it is decided by a comparison that has to be constructed carefully — which is where most of the work of measuring this actually goes.
Why the denominator is the hard part
The ratio is meaningless without a rigorous baseline, and the baseline is easy to fudge in the direction that flatters us.
The comparison group has to be equivalent code — the same functionality, produced from the same requirement, by a comparable process. If the baseline code is deliberately worse, the ratio inflates. If the baseline code was written by the most senior engineer on the team under ideal conditions, the ratio collapses and no tool ever wins.
The baseline must also be verified as equivalent before it can be timed. A third party confirms that the spec and the code cover the same set of behaviours. If they do not match, the comparison is void — the pair is discarded and regenerated. This is the step that gets skipped, and it is the step that makes the number mean something.
How we avoid measuring our own bias
Three rules, and they are the reason the number is worth anything.
Blind review. The reviewer does not know whether they are looking at a specification or code. Every review is labelled arbitrarily.
Counterbalanced assignment. Each reviewer sees the specification for one scenario and the code for another, with the mapping reversed for a second reviewer. A reviewer who systematically prefers one format is visible in the aggregate rather than hidden in the noise.
Randomised order. Within a reviewer’s set, the order of artifacts is randomised. Fatigue and learning effects are real and they do not favour either format systematically — but they do move the number, and an uncontrolled order turns that noise into bias.
What we count as review time
Net time from seeing the artifact to reaching a pass or reject decision. Environment waiting is excluded — it is not part of the review. Re-review after a rejection is tracked separately and not folded into the first measurement, because a rejected-then-revised artifact is measuring a different thing.
We report the median and a confidence interval, not the mean. Review times are heavy-tailed: one reviewer who took a coffee break moves the mean by a lot and the median barely at all.
The metrics we refuse to report
Scenarios executed. It moves when you add scenarios, including weak ones. It does not move when a bug ships.
Coverage percentage. The same problem, plus it can be gamed by the party who writes the tests. If the model writes both the spec and the tests, coverage measures agreement rather than correctness.
Lines of code generated. Not a quality signal. More code is not more value.
Time to first working build. A fast build of the wrong thing is fast.
Number of active users. For an early-access platform, this mostly measures how many people were willing to talk to us.
Each of these is easy to report and easy to move without the software improving. A metric you can move without the software getting better is not a measurement — it is a decoration.
The metrics that do earn their place
Assertion-strength distribution. The share of scenarios graded strong, normal, and weak. Reported next to coverage, always, because either one alone is misleading.
Escape rate. The fraction of real defects that a covering scenario did not catch. This is the only quality signal derived from observed reality rather than from inspection, and it is the hardest to move artificially.
Defect attribution distribution. The split between spec_gap, spec_error, implementation_defect, environment_defect, and user_error.
This one is strategically the most useful, because it tells you where to intervene:
- Mostly
spec_gap— verification is fine, the specification is thin. Intervene upstream. - Mostly
implementation_defect— the specification is good, the generation is not. Intervene in the model or the toolchain. - Mostly
spec_error— a misunderstanding is systematic. A style guide entry usually fixes it. - Mostly
user_error— the specification is ambiguous enough that a reasonable person read it wrong. That is a specification defect under a different label.
Environment reliability. Fraction of trial environments that became reachable without intervention, and mean time from build to reachable. This one matters more than it appears: an environment that flakes gets blamed on the generated code, and trust collapses in a way no amount of code quality recovers.
Cost per successful build, split by region. Not a vanity metric — it goes into the pricing model, and if it does not, the pricing is a guess.
The metric we watch most
How often a clause is approved without being read.
Not directly measurable, so we approximate it: approval velocity against clause count, the proportion of clauses with no rejection history, and the strength distribution of the clauses approved fastest.
A specification approved in ninety seconds is a specification nobody verified. The platform will not stop you — but it will show you the shape of what you did, in the same terms, every time.
Why publish this at all
Because a platform that reports only flattering metrics is asking to be trusted on faith, and we are asking for something more specific: we are asking people to try the workflow and check our arithmetic.
If the ratio is not below 0.45, the approach is wrong and we should say so. Publishing the measurement protocol — including the parts designed to make the number less flattering to us — is the only way to make the claim worth anything.
The measurement protocol is documented in the product documentation, and the underlying reasoning is in spec-driven development versus BDD, where the objections are taken seriously.