This page describes a problem we think is the most important unsolved problem in automated verification, and the mechanism we use to make it visible.
The problem
A test suite can pass while the software is wrong. This is not an edge case. It is the default failure mode of automated verification, and it happens for a simple reason: the specification and the tests are written by the same author.
When one party writes both the requirement and the check that the requirement is met, it can write a check that agrees with itself. A scenario that asserts only "the request succeeds" passes for every implementation that returns 200 — including the ones that return the wrong data, charge twice, or ignore the input entirely.
At scale, this produces a number that looks like success and means very little:
87% of scenarios verified.
If most of those scenarios assert almost nothing, the number describes the shape of your specification, not the quality of your software. It moves when you add weak scenarios, and it does not move when a real bug ships. It is a metric that cannot fail, which is what makes it useless.
The literature on automated agent evaluation reaches the same conclusion from a different direction: agents that follow the prescribed process can fail held-back cases that agents ignoring the process pass, because a process the agent can satisfy superficially is not the same as a process that checks the right things.
The mechanism
Every scenario receives a strength grade, derived from a static analysis of the scenario’s own steps. The grade is shown next to every coverage figure, and it is derived mechanically — never by asking a model to judge its own work.
The grades
Strong. The scenario asserts something that could fail for a realistic reason. It checks a specific value, a specific state transition, a specific error code. It has a negative branch, or it asserts on an object with several fields.
Normal. The scenario makes a real assertion, but a single one, or it checks a status code without checking anything about the body. It can catch gross errors. It will not catch a wrong amount.
Weak. The scenario asserts something that is true regardless of the implementation’s correctness. A bare status code. A Given with no state. A step that restates the request rather than checking the response. An unconditional skip.
What the static analysis looks for
- Tautological predicates. A
Thenthat restates theWhen— "the customer requests a refund" followed by "the customer requests a refund" — asserts nothing by construction. - Status-only assertions. A
Thenthat checks only that the call succeeded, with no assertion on the payload or the resulting state. - Unconditional skips and pendings. A scenario marked
skipwithout a reason, or markedpending, never runs and is usually counted as coverage anyway. - Missing negative branches. A feature whose scenarios all describe the happy path. This is not a weakness of any individual scenario, so it is graded at the feature level: a feature with no negative branch is capped at
normalregardless of how its individual scenarios score. - Assertion duplication. Two scenarios whose assertions are near-identical are counted once. Four copies of "returns 200" are one weak assertion, not four.
- Assertion breadth. How many distinct objects the assertions touch. A scenario asserting a single field from a ten-field response is narrow; one asserting status, amount, currency, and the resulting order state is broad.
Why it is static, and why it is not a model
The obvious alternative is to ask a model: "is this scenario a strong test?" We do not, and the reason is worth stating because it is one of the more counterintuitive findings in this area.
Self-assessment rates collapse as models get stronger. In a 90-agent trial, self-verification rates ranged from 100% for the strongest models to 0% for a weaker model family. A system that gets better at writing code becomes worse at honestly judging its own work — because it can produce a plausible-looking justification for almost any assertion. Self-assessment is not merely unreliable; it is inversely correlated with the capability it is supposed to measure.
So the grading is mechanical, and it is mechanical on purpose: it can be wrong in narrow, predictable ways, and its narrowness is visible. A static analyser that cannot read semantics will occasionally grade a genuinely good scenario as normal. That is an acceptable error. A model that cannot read semantics will confidently grade a bad scenario as strong, and that is not.
What it does not claim
The grade is not a measure of coverage quality in the abstract, and we do not present it as one. It is a measure of whether a specific scenario, as written, can fail for a realistic reason.
The strongest available evidence of a weak scenario is empirical: a real defect that the scenario did not catch. So the platform also tracks escape rate — defects found in the trial environment that a scenario covering the same clause should have caught. A scenario whose escape rate is non-zero is downgraded regardless of its static grade.
How it appears in the completion map
Coverage and strength are always shown together. Never one without the other.
The completion map reports, per clause:
- Whether the clause is covered by an executed scenario
- The strength grade of the covering scenario
- Whether any real defect escaped that coverage
- Whether the clause is covered only by a degraded scenario
Which means the map can express "this is done" and "this is covered but the coverage is weak" as different states — and it does, because collapsing them into one percentage is how the whole exercise goes wrong.
What you should do with the grades
Treat weak scenarios as missing tests. A weak scenario is not a passing test; it is a comment written in the shape of a test. It should be rewritten or removed, not counted.
Require a negative branch per feature, not per scenario. If a feature has no scenario describing what must not happen, that feature is not specified. This single rule catches more real defects than any other convention we know of.
Watch the strength distribution, not the coverage number. A specification that is 90% covered and 60% strong is in better shape than one that is 70% covered and 95% strong — and the coverage number alone will tell you the opposite.
Let escape rate drive the queue. The scenarios that let real defects through are the ones worth your attention next. That list is short, and it is derived from evidence rather than from a heuristic.
The honest summary
We do not claim the platform solves this. A weak scenario is still a weak scenario, and grading it does not make it strong — it only makes it visible, which is a precondition for fixing it.
What we do claim is narrower and more defensible: the platform will not let a coverage number stand alone. If your specification is 90% covered, you will see how much of that coverage is real. Any tool that reports a single coverage percentage without a strength qualifier is reporting a number that can be moved without the software improving at all.