An Instruction Is Not an Invariant
The prompt said scenarios must be distinct. It said so the entire time the model was returning five overlapping ones. And the control we shipped for multi-step cases turned out never to reach the model at all.
The people building Hawzu
Notes from the team building Hawzu — how it works inside, and what we have learned about running QA well.
The prompt said scenarios must be distinct. It said so the entire time the model was returning five overlapping ones. And the control we shipped for multi-step cases turned out never to reach the model at all.
Ask most QA assistants whether a release is ready and you get the general opinion of the internet. The useful answer depends on what your team wrote down — which means the retrieval matters more than the model.
Observatory's insights carry their question as a first-class field, and the scorecard reports how much of your automation your builds have actually run — not how much your test cases claim.
You open a release you signed off in March. The numbers are different now. Nothing was tampered with — the world simply kept moving, and the report kept recalculating.
Drawing a line between a requirement and a test case is the easy part. Traceability is what happens to that line six months later, when the requirement has children and the defect no longer points at a step.
Every team builds a QA dashboard in the first month and stops opening it by the third. The charts aren't wrong — nobody ever decided what any of them was supposed to change.
A QA report that tries to serve the whole company serves nobody. The useful question isn't what to include — it's who is reading, what they'll do next, and whether the number will still say the same thing next month.
Almost every QA team starts in a spreadsheet, and most stay longer than they meant to. The interesting question isn't whether Excel is 'wrong' — it's which specific thing breaks first, and what you lose on the way out.
UAT has different testers, a different goal and a different output than QA testing. You rarely need a dedicated UAT feature to run it well — you need to know which ordinary parts to assemble, and where the fit genuinely stops.
Most defect workflows fail in one of two directions: too many states nobody maintains, or too few to answer 'who has this now?'. The interesting decisions are about ownership and handover, not the diagram.
"We're 80% executed" is the number everyone reports and almost nobody can act on. Tracking a cycle well means knowing what's left, who holds it, and which results aren't finished being results.
Someone deletes forty obsolete test cases on a Friday. The interesting question isn't whether they're gone — it's what happened to every run that ever exercised them.
Someone improves the login sequence. Three hundred test cases use it and forty runs are in flight. Reuse is the easy half; the version is the half that makes reuse safe.
Test documentation isn't written for the person who wrote it. It's written for whoever runs it in eight months — and almost every decision about detail, structure and reuse follows from taking that seriously.
Every regression suite grows and almost none of them shrink. The management problem isn't writing the tests — it's deciding what belongs, what runs this cycle, and what to do about the ones nobody trusts.
A tester should be able to run tests without being able to reshape the run. That sentence is trivial to say and impossible to express as view, create, edit or delete.
One status field usually ends up doing two jobs: recording what a human decided, and implying what the data found. Those need to be different fields, and only one of them should be typeable.
You reopen last month's run to fix one result and the server refuses. That refusal is the feature — but what it refuses, and what it deliberately still allows, took more thought than the lock itself.
"Checkout is broken." Every QA team has filed that defect and every developer has bounced it. The information wasn't missing — it was on screen, and the form didn't ask for it.
Someone edits a test case. What happens to the run that used it last week? The answer decides whether your execution history is evidence or decoration.