An Instruction Is Not an Invariant

The prompt said the scenarios must be distinct.

It said so clearly, in capitals, near the top. It said so the entire time the model was returning five scenarios that were obviously variations of the same one — and nobody noticed for a while, because the output looked like five scenarios and counting to five is not a review.

When we went looking for why, the cause turned out to be sitting one line further down. The request handed the model a batch of sections from the source document and asked for almost exactly as many scenarios. Asking for as many scenarios as there are sections reads as an instruction of its own: turn each section into a scenario. The distinctness rule was arguing with a number, and losing.

That’s the sentence this whole post exists for. An instruction in a prompt is a request. It is not a constraint, and building as though it were is how AI features quietly stop doing what you think they do.

The rule that never reached the model

There is a worse failure than an instruction being ignored, and we shipped it.

Hawzu lets you choose how generated cases are written. The setting behind that choice offered multi-step and single-step cases as well as Gherkin, and it only ever changed anything for Gherkin. Asking for multi-step and asking for single-step sent the model exactly the same request. Someone choosing multi-step was sending a preference no model ever saw, and the screen gave every appearance that it had.

Reading through the setup did not find this. Counting the output did. Different models handed the same request split between single-step and multi-step in wildly different proportions, and that spread is the tell. When a setting works, models differ over how well they follow it; when it’s missing, they differ over everything, each falling back on its own habits.

The fix was small, and the step setting now works. The thing that stops it recurring is not small: every step choice must now produce its own distinct request, and that is checked automatically rather than assumed by whoever changes it next.

What can be built in, and what can’t

Some rules don’t have to be obeyed, because they can’t be broken.

Every time Hawzu asks for scenarios, it also tells the model the exact format the answer must take. The requirements a scenario may cite are part of that format: only the ones that came in with this request. The AI can only link a case to requirements you included. Not because it was told not to invent others, but because an answer naming any other one can’t be produced at all. When it later writes the full cases, it can’t touch those links: they carry over from the scenarios you already reviewed, so a link you approved can’t be quietly redone.

That is the strongest form a rule can take, and it has a limit. Evidence may get weaker between the two stages but never stronger — a case can’t claim better evidence than the scenario it came from. That is a rule about how two answers relate, and a format can only describe one answer at a time. So Hawzu checks it on the way through: a case that claims too much is lowered to match, and the correction is recorded rather than hidden.

Which is roughly where the line falls: the required format for what a single answer may contain, Hawzu’s own code for how answers relate, the prompt for everything left over. Every rule in that last group gets an automatic check, because that is where things go wrong silently.

enforced requested

1. Required format What one answer may contain
   A case can only link to requirements you included fails: It can’t — no other answer can be produced
2. Code How two answers relate
   Evidence may weaken between stages, never strengthen fails: Corrected on the way through, and recorded
3. Prompt Everything left over
   Scenarios must be distinct · each step choice changes the request fails: Silently — unless something checks it

No evidence, no test

Ask a model to generate test cases from a requirement and it will always produce test cases. That’s the trouble. Requirements describe the happy path and say nothing about the payment provider timing out, and a model handed one fills the gap from its general knowledge. The invented cases look exactly like the sourced ones — same format, same confidence, same tidy numbered steps.

The dangerous output isn’t the case that’s wrong. It’s the case that’s plausible and unsourced.

So generation isn’t allowed to return a bare test case. Every item comes back labelled with where it came from, and there are exactly three labels. Documented — the behaviour is stated in the source. Derived — it follows from what’s stated, by ordinary reasoning. Unknown — the model wants to test something the material doesn’t cover.

The first two become test cases. The third becomes a documentation gap: in Hawzu it comes back in its own list, never as a draft with a tick box beside it, because the moment gaps share a list with drafts somebody ticks one. What began as “the model wants to test what happens when the payment provider times out” turns into “your requirement doesn’t say what happens when the payment provider times out”, which is more useful than the test would have been.

Three labels, not a confidence score. A number invites an argument about where to draw the line; three named labels force the decision at generation time. Which labels may become test cases is decided in exactly one place, so a fourth label added later can’t quietly slip into the pile you can import.

That rule is checked automatically rather than left to the prompt, and the reason is our own history. While building this feature we shipped four prompt changes without measuring them, and “scenarios must be distinct” was in the prompt the whole time the model was ignoring it.

We shipped the model that scored lower

The benchmark had a winner. We didn’t ship it.

When we compared models on a set of real requirement documents, one came out ahead on quality. Before the comparison ran, we checked something the scores don’t capture: whether each model reliably sticks to a required answer format, or merely accepts one and then answers however it likes. The top scorer didn’t stick to it. Everything in the previous section — links only to requirements you included, the three evidence labels — would have slipped from a guarantee to a well-worded suggestion.

A small quality gap is worth less than the difference between a guarantee and a hope. So the rule chose the model, and the one in production is the one that scored lower.

Two things we removed, and why

The unglamorous half of building this was taking things out.

The review states came out. Generated cases used to arrive as drafts, with “in review” and “approved” waiting for them. Reasonable design, no reviewer — Hawzu has no review-and-approve flow, so nothing ever moved a case past draft. Cases sat as drafts permanently: a test case that works, labelled as one that doesn’t. The states went. Accepting a case is the decision, and accepting is a click.

A second model came out. Every model we tried covered what a requirement states well and what it implies badly — concurrency, partial failure, recovery. Picking a better model didn’t close that. Running two and merging their results did help in testing, and the reason was pleasingly perverse: the second model was weaker, and earned its place by being wrong in different directions.

In real use it made the output worse. Hawzu spots duplicate scenarios by how closely the wording of their titles matches. Two models describing one idea in their own words don’t match closely — “Insufficient Funds Before Execution” against “Fail execution after balance becomes insufficient”. Similar enough to be the same scenario, different enough to survive the filter. And because the merge gave the second model a share of the places rather than adding to the end, every restatement it got in pushed out something real. On the first document we tried, four restatements displaced four genuinely distinct scenarios, one of them “Prevent duplicate payment submission”.

No cut-off fixes that, because genuinely different scenarios match just as closely by that measure. So the second model is switched off, the reason is written down beside the switch, and so is the way back: comparing titles by what they mean rather than by how they’re worded.

What we kept is the one measurement that isn’t a stand-in. Every accepted case remembers which generation produced it, and keeps that through edits and deletes. Acceptance rate is the only number that says whether generation is useful; benchmark scores and ratings measure something somebody chose as a substitute.

Key takeaways

• Prompts are requests. Anything that must hold needs an automatic check, because a broken prompt rule produces plausible output rather than an error.
• Check that your instruction reaches the model at all. One of ours didn’t, and only counting the output revealed it.
• Prefer rules the required format enforces. Where the format can’t express the rule — anything relating two answers — enforce it in code and say so.
• Make the model declare its evidence for every item, from a small fixed set of labels. Unsourced output is a finding about your documentation, not a draft to review.
• Gains measured in testing don’t survive real use if the live product compares results differently from the way your testing judged them.

None of this makes the model more accurate. It makes the system honest about which outputs have a source — the same commitment as keeping Oracle’s answers traceable to the documents they came from, applied at the point where AI writes into your repository instead of just talking to you.

A rule with no failure mode isn’t a rule. It’s a hope with good formatting.