The value of a trial environment is not that it is pleasant to use. It is that it surfaces things the automated checks did not, and turns them into corrections to the specification.
Findings
A finding is anything you observed in the trial environment that contradicts the specification. You attach it to the clause it should have been caught by.
Attachment matters. A finding attached to a clause does three things:
- It regenerates that clause with the failure attached, so the fix lands in the specification and not just in a ticket.
- It downgrades the covering scenario’s grade, because a scenario that let a real defect through is not a strong scenario any more.
- It feeds the escape-rate metric, which is the only quality signal in the system derived from observed reality rather than from static analysis.
A finding you cannot attach to a clause is still recorded. It is reported as a specification gap, and it appears in the completion map as a clause that should exist and does not. That is a more useful outcome than it sounds: unattachable findings are exactly the requirements you never wrote down.
Defect attribution
Every finding is attributed to a cause:
| Cause | Meaning | Fix |
|---|---|---|
spec_gap |
The requirement was never written down | Add a clause |
spec_error |
The requirement was written down, but wrongly | Regenerate the clause |
implementation_defect |
The requirement was right; the implementation is wrong | Fix the implementation |
environment_defect |
The environment misbehaved — seed data, configuration, dependency | Fix the environment |
user_error |
The behaviour is correct; the expectation was wrong | Close with a reason |
The distribution across these five categories is the most informative output the platform produces, and it is the one we watch most closely.
Why the distribution matters more than the count
A hundred findings that are all spec_gap is a different situation from a hundred findings that are all implementation_defect, and the difference is strategic.
Mostly spec_gap means the verification is fine and the specification is thin. The fix is upstream: write more clauses, or write them at an earlier stage. The platform is working as intended — it is telling you the requirements were not captured.
Mostly implementation_defect means the specification is good and the generation is not. The fix is downstream: the model is producing code that does not satisfy a correct specification. This is the case where you would want to change the implementation approach rather than the specification.
Mostly spec_error means the specification is being written with a misunderstanding — usually about a default, a boundary, or a piece of domain vocabulary. The fix is usually a style guide entry, and it usually pays off across the whole project.
Mostly user_error is a warning sign about the platform, not about you. It means the specification was ambiguous enough that a reasonable person read it the wrong way. That is a specification defect wearing a different label, and it should be reclassified.
The platform computes the distribution per project and per clause type, and surfaces it in the completion map.
Escape rate
Escape rate is the fraction of real defects that a scenario covering the same clause did not catch.
It is the only quality metric in the system grounded in observed reality. Static analysis tells you a scenario looks weak; escape rate tells you that it was weak, because something got through.
Scenarios with a non-zero escape rate are downgraded and highlighted. They are the highest-value targets in the project, because each one is a place where the specification claimed coverage and did not deliver it.
Over time, the ideal is not that escape rate reaches zero. It is that the scenarios with a non-zero escape rate are a small, visible, and shrinking list — and that fixing them improves the specification’s overall fidelity, not just the count.
The loop, closed
trial finding
│
▼
attach to clause ──▶ regenerate clause ──▶ approve ──▶ rebuild ──▶ re-trial
│
▼
downgrade covering scenario's grade
│
▼
update escape rate and defect attribution
The point of the loop is that it terminates in the specification. A finding that lives only in a ticket tracker has fixed a bug; a finding that lands back in the clause has fixed a class of bug.
What we deliberately do not do
We do not auto-close findings. A finding that resolves itself on the next build is not the same as a finding that was correctly attributed. The platform can suggest "this appears resolved by clause X" — it does not apply it.
We do not treat finding count as a quality score. More findings is not worse. A team that finds twenty problems in a trial has done the trial properly; a team that finds none has either written an excellent specification or has not looked.
We do not let findings become a permanent backlog. A clause with an open finding stays visible in the completion map, but a project with a growing set of open findings has a problem that is better solved by writing a better specification than by triaging tickets.