Managing Test Cases Doesn't Answer the Real Question
Teams have gotten good at managing test cases — but doing it well still doesn't answer the one question that matters: are we ready to ship?
Essays, engineering notes and practical guides for QA teams — from the people building Hawzu.
23 posts
No posts match. Try a different word or category.
Teams have gotten good at managing test cases — but doing it well still doesn't answer the one question that matters: are we ready to ship?
Every release meeting ends at one question: are we ready to ship? No single metric answers it. Confidence isn't read off a number — it's assembled from connected evidence, then judged by people.
We want software to tell us when it's ready. It can't. Every report, dashboard, and test run is just an attempt to reduce uncertainty enough to decide.
The prompt said scenarios must be distinct. It said so the entire time the model was returning five overlapping ones. And the control we shipped for multi-step cases turned out never to reach the model at all.
Ask most QA assistants whether a release is ready and you get the general opinion of the internet. The useful answer depends on what your team wrote down — which means the retrieval matters more than the model.
Observatory's insights carry their question as a first-class field, and the scorecard reports how much of your automation your builds have actually run — not how much your test cases claim.
You open a release you signed off in March. The numbers are different now. Nothing was tampered with — the world simply kept moving, and the report kept recalculating.
Drawing a line between a requirement and a test case is the easy part. Traceability is what happens to that line six months later, when the requirement has children and the defect no longer points at a step.
Someone deletes forty obsolete test cases on a Friday. The interesting question isn't whether they're gone — it's what happened to every run that ever exercised them.
Someone improves the login sequence. Three hundred test cases use it and forty runs are in flight. Reuse is the easy half; the version is the half that makes reuse safe.
A tester should be able to run tests without being able to reshape the run. That sentence is trivial to say and impossible to express as view, create, edit or delete.
One status field usually ends up doing two jobs: recording what a human decided, and implying what the data found. Those need to be different fields, and only one of them should be typeable.
You reopen last month's run to fix one result and the server refuses. That refusal is the feature — but what it refuses, and what it deliberately still allows, took more thought than the lock itself.
"Checkout is broken." Every QA team has filed that defect and every developer has bounced it. The information wasn't missing — it was on screen, and the form didn't ask for it.
Someone edits a test case. What happens to the run that used it last week? The answer decides whether your execution history is evidence or decoration.
Every team builds a QA dashboard in the first month and stops opening it by the third. The charts aren't wrong — nobody ever decided what any of them was supposed to change.
A QA report that tries to serve the whole company serves nobody. The useful question isn't what to include — it's who is reading, what they'll do next, and whether the number will still say the same thing next month.
Almost every QA team starts in a spreadsheet, and most stay longer than they meant to. The interesting question isn't whether Excel is 'wrong' — it's which specific thing breaks first, and what you lose on the way out.
UAT has different testers, a different goal and a different output than QA testing. You rarely need a dedicated UAT feature to run it well — you need to know which ordinary parts to assemble, and where the fit genuinely stops.
Most defect workflows fail in one of two directions: too many states nobody maintains, or too few to answer 'who has this now?'. The interesting decisions are about ownership and handover, not the diagram.
"We're 80% executed" is the number everyone reports and almost nobody can act on. Tracking a cycle well means knowing what's left, who holds it, and which results aren't finished being results.
Test documentation isn't written for the person who wrote it. It's written for whoever runs it in eight months — and almost every decision about detail, structure and reuse follows from taking that seriously.
Every regression suite grows and almost none of them shrink. The management problem isn't writing the tests — it's deciding what belongs, what runs this cycle, and what to do about the ones nobody trusts.