The assertion strength problem

We are going to describe a mechanism we built, and explain why the alternative obvious implementation — asking a model to judge the work — does not work. The reason is genuinely surprising, and it is the reason this is worth writing down.

The problem, stated plainly

A test suite can pass while the software is wrong. This is not a rare edge case. It is the default failure mode of automated verification, and it has a simple cause: the specification and the tests are written by the same author.

When one party writes both the requirement and the check that the requirement is met, it can write a check that agrees with itself. A scenario that asserts only "the request succeeds" passes for every implementation that returns 200 — including the ones that return the wrong data, charge the customer twice, or ignore the input entirely.

Scaled up, this produces the number that passes for quality everywhere in software:

87% of scenarios verified.

If most of those scenarios assert almost nothing, the number describes the shape of your specification, not the quality of your software. It moves when you add weak scenarios. It does not move when a real bug ships. It is a metric that cannot fail, which is exactly what makes it useless.

The evidence

This is not a new observation, and it has been measured carefully.

In an evaluation of 90 coding agent trials, self-verification rates ranged from 100% for the strongest models down to 0% for a weaker model family. The direction of the relationship is the striking part: self-assessment gets worse as the model gets better at the underlying task. A model that can produce a plausible implementation can also produce a plausible-sounding justification for almost any assertion. Self-review is not merely unreliable — it is inversely correlated with the capability it is supposed to measure.

The same body of work contains a warning that matters even more for our case. When the agent is allowed to write its own tests, a suite that passes is not evidence that the implementation is correct. In one experiment, agents that followed the prescribed process failed held-back test cases, while agents that ignored the process entirely passed. The process was satisfiable without satisfying the thing the process was for.

If the agent writes both the spec and the tests, it is writing both halves of that arrangement.

What we do instead

Every scenario gets a strength grade, derived by static analysis of the scenario’s own steps. Never by asking a model.

Strong. The scenario asserts something that could fail for a realistic reason: a specific value, a specific state transition, a specific error code, several fields at once.

Normal. The scenario makes a real assertion, but a single one, or checks a status code without checking the body. It catches gross errors. It will not catch a wrong amount.

Weak. The scenario asserts something true regardless of the implementation’s correctness. A bare status code. A Given with no state. A step that restates the request. An unconditional skip.

The analysis looks for tautological predicates, status-only assertions, unconditional skips, missing negative branches, duplicate assertions across scenarios, and narrow assertion breadth. It is deliberately shallow — it does not attempt to read semantics.

And that shallowness is the point. A static analyser will occasionally grade a genuinely good scenario as normal. That is an acceptable error: it is narrow, predictable, and visible. A model judge will confidently grade a bad scenario as strong, and that is not acceptable, because it is invisible and it gets worse as the models improve.

The empirical correction

Static analysis is a prior, not a verdict. So there is a second signal, and it is the only quality metric in the system that comes from observed reality rather than from inspection.

Escape rate: the fraction of real defects that a scenario covering the same clause did not catch.

A scenario that let a real defect through is downgraded, regardless of its static grade. That is a much harder signal to fake, because it requires a defect to actually escape.

Over time the two signals converge, and the escape-rate list becomes the shortest, highest-value queue in the project — a short list of places where the specification claimed coverage and did not deliver it.

How it appears in the product

Coverage and strength are always shown together. Never one without the other.

The completion map reports, per clause: whether it is covered by an executed scenario, the strength grade of that scenario, whether any real defect escaped it, and whether it is covered only by a degraded scenario.

Which means the map can express "this is done" and "this is covered, but the coverage is weak" as different states. Collapsing them into one percentage is how the whole exercise goes wrong.

What we are not claiming

We do not claim this solves the problem. A weak scenario is still weak, and grading it does not fix it — it only makes it visible, which is a precondition for fixing it.

What we do claim is narrower and, we think, defensible: the platform will not let a coverage number stand alone. If your specification is 90% covered, you will see how much of that coverage is real. Any tool that reports a single coverage percentage without a strength qualifier is reporting a number that can be moved without the software improving at all.

The detailed mechanism is documented in assertion strength.