An Instruction Is Not an Invariant
The prompt said scenarios must be distinct. It said so the entire time the model was returning five overlapping ones. And the control we shipped for multi-step cases turned out never to reach the model at all.
13 posts
The prompt said scenarios must be distinct. It said so the entire time the model was returning five overlapping ones. And the control we shipped for multi-step cases turned out never to reach the model at all.
Ask most QA assistants whether a release is ready and you get the general opinion of the internet. The useful answer depends on what your team wrote down — which means the retrieval matters more than the model.
Observatory's insights carry their question as a first-class field, and the scorecard reports how much of your automation your builds have actually run — not how much your test cases claim.
We want software to tell us when it's ready. It can't. Every report, dashboard, and test run is just an attempt to reduce uncertainty enough to decide.
You open a release you signed off in March. The numbers are different now. Nothing was tampered with — the world simply kept moving, and the report kept recalculating.
Drawing a line between a requirement and a test case is the easy part. Traceability is what happens to that line six months later, when the requirement has children and the defect no longer points at a step.
Someone deletes forty obsolete test cases on a Friday. The interesting question isn't whether they're gone — it's what happened to every run that ever exercised them.
Someone improves the login sequence. Three hundred test cases use it and forty runs are in flight. Reuse is the easy half; the version is the half that makes reuse safe.
A tester should be able to run tests without being able to reshape the run. That sentence is trivial to say and impossible to express as view, create, edit or delete.
One status field usually ends up doing two jobs: recording what a human decided, and implying what the data found. Those need to be different fields, and only one of them should be typeable.
You reopen last month's run to fix one result and the server refuses. That refusal is the feature — but what it refuses, and what it deliberately still allows, took more thought than the lock itself.
"Checkout is broken." Every QA team has filed that defect and every developer has bounced it. The information wasn't missing — it was on screen, and the form didn't ask for it.
Someone edits a test case. What happens to the run that used it last week? The answer decides whether your execution history is evidence or decoration.