Cross-format consistency

A specification has two halves that describe the same system from different angles. When they disagree, the disagreement is invisible to any tool that looks at only one of them — and it is one of the most expensive kinds of defect to find late, because it usually surfaces as a confusing integration failure rather than as an obvious mistake.

This is the single most valuable interception point in the specification stage, and the platform implements it as a dedicated verification layer.

What is checked

Referenced operations must exist

Every scenario that calls an interface names it with an operationId. If the scenario references requestRefund and the document does not define it, the scenario can never run.

Scenario: Refunding a settled order
  When the customer requests a refund through operation "requestRefund"
  Then the refund amount is 42.00 EUR
# paths:
#   /orders/{orderId}/refund:
#     post:
#       operationId: refundOrder      # ← mismatch: requestRefund

Caught: the scenario references an operation that does not exist.

Referenced status codes must be declared

A scenario that expects a 422 response requires the document to declare a 422 response for that operation, and the example values in the scenario must satisfy the schema of that response.

A scenario expecting a response the contract does not describe is either a missing contract or a wrong expectation. Both are defects, and they are indistinguishable from the outside — which is precisely why they need to be caught here.

Example values must satisfy their schema

Where a scenario carries a concrete payload, the values are validated against the relevant schema. A scenario asserting amount: 42.0 against a schema requiring amount to be a string is caught here rather than as a type error three layers down.

Every operation should be reachable

The check runs in reverse as well: operations defined in the document that no scenario touches are listed as uncovered. Not an error — an observation, and a valuable one, because an operation nobody exercised is an operation nobody checked.

Tags and operationIds must line up

Where scenarios are tagged with an operation reference, the tag must resolve. A typo in a tag silently detaches a scenario from the contract it was meant to verify, which is worse than not tagging it at all.

Why this cannot be delegated to existing tools

Each individual check has a home in the existing toolchain. Spectral validates a document. schemathesis tests a live API against a document. gherkin-lint checks a feature file. What none of them does is compare the two documents against each other, because that requires understanding that a string in one file refers to an identifier in the other.

That is the operation the platform adds. It is not a replacement for the existing tools — it runs them and adds the join.

Why this matters more than it looks

Consider what a missing 422 response costs.

Without the check: the specification is approved, the build passes, the trial environment deploys. A tester triggers the out-of-window refund. The service returns 500, because the handler for "refused" was never written — the document never mentioned it, so the draft never produced it. The tester files a defect. Somebody reads the spec and finds a perfectly reasonable clause saying the refund is refused outside the window, with no error code attached to it.

The defect is real, the spec was approved, and the spec was the source of the ambiguity. The cost is not the bug; it is that the review passed something that was never actually pinned down.

With the check: the build fails at the consistency layer, names the operation and the missing status code, and attributes it to the clause. The review never gets to approve it.

What the platform reports

Failures are reported at clause level, not file level. A consistency failure names:

  • The scenario and the clause it came from
  • The operation and the document section involved
  • What was expected and what was found
  • Whether the likely cause is a missing contract, a wrong scenario, or a wrong expectation

That last distinction is the useful one. "The document does not declare a 422 response for requestRefund" is actionable immediately; "consistency error" is not.

Related: assertion strength covers the other half of the problem — scenarios that are internally consistent but too weak to catch anything.

Next