An Instruction Is Not an Invariant

The prompt said the scenarios must be distinct.

It said so clearly, in capitals, near the top. It said so the entire time the model was returning five scenarios that were obviously variations of the same one — and nobody noticed for a while, because the output looked like five scenarios and counting to five is not a review.

When we went looking for why, the cause turned out to be sitting one line further down. The request retrieved about thirty-two sections from the source document and asked for thirty scenarios. That is a one-to-one ratio, and a one-to-one ratio reads as an instruction: turn each section into a scenario. The distinctness rule was arguing with a number, and losing.

That’s the sentence this whole post exists for. An instruction in a prompt is a request. It is not a constraint, and building as though it were is how AI features quietly stop doing what you think they do.

The rule that never reached the model

There is a worse failure than an instruction being ignored, and we shipped it.

Hawzu lets you ask for multi-step cases or single-step ones. The setting reached the prompt builder and was consumed by exactly one branch — the one that handles Gherkin. For the other two modes it did nothing, and the prompts for “multiple” and “single” came out byte-identical. Someone choosing multi-step was sending a preference no model ever saw, and the interface gave every appearance that it had.

Reading the code did not find this. Counting did: across 918 generated cases, 56% came back single-step, at rates ranging from 22% to 89% between models handed the same prompt. That spread is the tell. When a control works, models differ over how well they follow it; when it’s absent, they differ over everything, each falling back on its own habits.

The fix is one line. The thing that stops it recurring is not — three step modes must now produce three distinct prompts, asserted in the test suite rather than assumed by whoever edits the builder next.

What the schema can hold, and what it can’t

Some rules don’t have to be obeyed, because they can’t be broken.

Every generation request builds its own JSON schema. The requirement identifiers that came in on that request become an enum on the field where the model cites what it’s covering. A model cannot attribute a case to a requirement that doesn’t exist in your project — not because it was told not to, but because the constrained decoder cannot emit the token. The second stage goes further and removes the attribution field altogether, inheriting what the first stage already had reviewed, so an attribution you approved can’t be silently re-derived.

That is the strongest form the rule can take, and it has a ceiling. Evidence strength may weaken between the two stages but never strengthen — a case can’t claim better grounding than the scenario it came from. That is a relationship between two fields, and strict JSON schema has no way to say it. So it lives in a clamp in code, applied on the way through, returning both the corrected value and the fact that it corrected something.

Which is roughly where the line falls: the schema for what a single field is allowed to be, code for relationships between fields, the prompt for everything left over — and a test for every rule in that last category, because that is the category where things go wrong silently.

No evidence, no test

Ask a model to generate test cases from a requirement and it will always produce test cases. That’s the trouble. Requirements describe the happy path and say nothing about the payment provider timing out, and a model handed one fills the gap from its priors. The invented cases look exactly like the grounded ones — same format, same confidence, same tidy numbered steps.

The dangerous output isn’t the case that’s wrong. It’s the case that’s plausible and unsourced.

So generation isn’t allowed to return a bare test case. Every item comes back labelled with where it came from, and there are exactly three labels. Documented — the behaviour is stated in the source. Derived — it follows from what’s stated, by ordinary reasoning. Unknown — the model wants to test something the material doesn’t cover.

The first two become test cases. The third becomes a documentation gap: in Hawzu it comes back in its own list, never as a draft with a tick box beside it, because the moment gaps share a list with drafts somebody ticks one. What began as “the model wants to test error handling on this endpoint” turns into “your requirement doesn’t say what happens when this endpoint errors”, which is more useful than the test would have been.

Three values, not a confidence score. A number invites an argument about the threshold; three named states force the decision at generation time. The routing between them is one named set rather than a comparison repeated at each call site, so a fourth label added later can’t quietly default into the importable pile.

That rule is asserted in the test suite, and the reason is written at the top of the file: four prompt edits shipped unmeasured in the session that built this feature, and “scenarios must be DISTINCT” was in the prompt the whole time the model was ignoring it.

We shipped the model that scored lower

The benchmark had a winner. We didn’t ship it.

Across twenty requirement documents one model came out ahead on quality, 0.722 against the 0.656 of the one now in production. Before the comparison ran, a canary probe checked something the scores don’t capture: whether each model’s provider actually enforces a supplied JSON schema, or merely accepts one. The top scorer didn’t. Everything in the previous section — the identifier enum, the three-value label, the guarantee that a citation is real — would have degraded from a structural property into a well-worded suggestion.

A seven-point quality gap is worth less than the difference between a constraint and a hope. So the invariant chose the model.

Two things we removed, and why

The unglamorous half of building this was taking things out.

A review status came out. Generated cases used to arrive as draft, with in_review and approved waiting for them. Reasonable design, no reviewer — the product has no review-and-approve flow, so the only writer was AI acceptance and there was no reader. Cases sat at draft permanently: a test case that works, in a state announcing it doesn’t. The states went. Acceptance is the decision, and acceptance is a click.

A second model came out. Every model we measured covered what a requirement states well, 76–98%, and what it implies badly, 17–35% — concurrency, partial failure, recovery. Picking a better model didn’t close that; the best solo score was 40.2%. Running two and merging did, 34.6% to 42.9%, and the reason was pleasingly perverse: the second model was weaker, and earned its place by being wrong in different directions.

In live use it made the output worse. Deduplication compares titles as strings, and two models describing one idea in their own words score 0.43 to 0.71 — “Insufficient Funds Before Execution” against “Fail execution after balance becomes insufficient”. Similar enough to be the same scenario, different enough to survive the filter. And because the merge reserves slots rather than appending — an ensemble that changes nothing is worse than no ensemble — every restatement admitted evicted something real. On the first document we tried, four restatements displaced four genuinely distinct scenarios, one of them “Prevent duplicate payment submission”.

No threshold fixes it, because 0.43 is also what truly different scenarios score. The flag is off, the reason sits next to it, and so does the way back: compare titles by embedding rather than by characters.

What we kept is the one measurement that isn’t a proxy. An accepted case carries the generation that produced it as a plain field, surviving edits and deletes and queryable beside status. Acceptance rate is the only number that says whether generation is useful; benchmark scores and judge ratings measure a proxy somebody chose.

Key takeaways

• Prompts are requests. Anything that must hold needs a test, because a violated prompt produces plausible output rather than an error.
• Check that your instruction reaches the model at all. Ours didn’t, for one setting, and only counting 918 outputs revealed it.
• Prefer constraints the decoder enforces. Where the schema can’t express the rule — anything relating two fields — enforce it in code and say so.
• Make the model declare its grounding per item, in a small closed set. Unsourced output is a finding about your documentation, not a draft to review.
• Offline gains computed against a semantic judge don’t survive contact with a pipeline that compares strings.

None of this makes the model more accurate. It makes the system honest about which outputs are grounded — the same commitment as keeping Oracle’s answers traceable to the documents they came from, applied at the point where AI writes into your repository instead of just talking to you.

A rule with no failure mode isn’t a rule. It’s a hope with good formatting.