Test Execution Tracking: What "80% Executed" Doesn't Tell You

Test execution tracking is knowing where a testing cycle actually stands while it’s still running — what’s been executed, what’s left, who holds it, what’s stuck, and whether the date you promised is still real.

It’s a different job from managing test cases, and the difference is easiest to see as two questions:

Test case management asks: what can we test?

Test execution tracking asks: what did we actually test, what’s left, what’s blocking it, and will we finish in time?

Most teams have a good handle on the first and report the second as a single percentage, which is where the trouble starts.

Why “80% executed” isn’t a status

Almost every cycle gets reported as a completion percentage, and it’s close to useless on its own for three reasons.

It’s a biased sample. Testers do the quick, well-understood cases first — not out of laziness, but because those are the ones you can knock out between meetings. The remaining 20% is disproportionately the slow, fiddly, blocked and ambiguous work. It’ll take longer than the first 80% did.

It hides who’s holding it. Eighty percent done with the last fifth spread across four people is a cycle that finishes. The same number with one person holding all of it is a cycle that doesn’t, and the percentage looks identical.

It says nothing about what’s stuck. Ten cases blocked on an environment aren’t “not done yet”. They’re not going to be done at all until someone fixes something, and no amount of testing time helps.

Statuses that carry information

The status vocabulary is where most tracking either works or quietly fails, and the two distinctions that matter most are the ones teams collapse.

Blocked is not Failed

Failed means the software did the wrong thing — that’s a finding, and it should produce a defect. Blocked means the tester couldn’t get far enough to find out: the environment is down, the test data doesn’t exist, an integration is returning 500s, they don’t have the permission the case requires.

Collapsing blocked into failed corrupts everything downstream. Your defect list fills with things that aren’t defects, your pass rate drops for reasons unrelated to quality, and — worst — the actual blockers become invisible, because they look like ordinary test failures rather than the shared impediment they are.

Not Executed is not zero

For manual cycles, Not Executed means “not yet”. For automated runs it can mean something much more interesting: the expected case never produced a result. That’s the difference between this passed and this never ran, and it’s one of the most common ways an automation suite quietly stops covering something. If a case disappears from the pipeline, an explicit Not Executed is what tells you — provided the status survives the import.

A result can be stale

Here’s the case that most often has nowhere to live. A test case is edited after it was executed in an open cycle — someone corrects a step, tightens an expected result. The pass recorded yesterday was against different instructions. It isn’t wrong exactly, but it’s no longer evidence for the case as it now reads. Whether a tool can even represent that state depends on whether a run holds a copy of the case or a pointer to it.

That state is neither “executed” nor “not executed”: it’s executed, and unfinished. Teams that have no way to express it either leave a misleading pass in place or wipe the result and lose the fact that someone did the work.

What to track instead

Remaining work, by owner

The useful burndown isn’t executed-over-time, it’s remaining over time against the days left. And it should be decomposable by person, because that’s the level at which you can actually do something — reassign, pair up, cut scope.

The signal worth alerting on is concentration: when one person holds a large share of what’s left, the cycle’s completion date is really that person’s calendar, and nobody has noticed.

Work that’s been assigned and not touched

Assigned-but-untouched is different from unassigned. Unassigned work is visible and gets picked up. Assigned work has an owner, so everyone assumes it’s in hand — right up until you discover that person has been on another project for a week.

Blockers, as their own list

Blocked cases should be a list somebody owns and works, not rows scattered through a grid. They’re usually a handful of root causes affecting many cases, and they’re the highest-leverage thing to fix mid-cycle: one environment repair can unblock thirty tests.

Pace, honestly

If you’ve executed 120 cases in four days and 300 remain, you need ten more days, not the four you have. Teams routinely know this implicitly and report “on track” anyway, because the completion percentage looks reassuring and nobody did the division.

How this works in Hawzu

Runs come in two shapes, and the distinction is deliberate. A standalone test run lives in a project and tracks work that isn’t tied to a release. A release execution belongs to a release and contributes to its progress and readiness. The docs are blunt about the consequence: testing recorded in a standalone run doesn’t reach the release dashboard, however thorough it was. Choose accordingly.

Testers set one of five statuses — Passed, Failed, Blocked, Skipped, Not Executed — and results are recorded per step as well as per case, with attachments landing on the step, so the evidence sits next to the observation rather than in a pile at the case level.

There’s a sixth status, Needs Rerun, that no user can set: the API rejects the attempt outright. It’s applied by the system when a test case is edited after it was pulled into an open run, marking the recorded result as stale. It’s also deliberately unreachable from automation — whether a result still counts is a human judgement, not something a CI report can assert. In the analytics it sits in an unusual place on purpose: executed and unfinished at the same time, so it counts as work done but is excluded from anything conclusive, and the burndown burns down Not Executed and Needs Rerun together.

A CI import can’t erase a gap. Importing results applies them only to rows the report actually matched; report rows that match nothing in the run are recorded separately and left inert. So a case the report never mentions keeps the status it had — it stays Not Executed rather than being quietly skipped or dropped, which is what makes the missing result visible at the end of an automated run.

Work is tracked per test case per data set, so a case running against four datasets is four independently assignable, independently statused rows. You can assign individually or in bulk, filter the run’s list by assignee including Unassigned, and assignment raises an in-app notification for the person receiving it.

The run’s analysis view is built around remaining work rather than pass rate: active executors, unassigned remaining, stale assigned remaining (work that has an owner and hasn’t been touched), each person’s share of what’s left, and an explicit concentration risk flag when one person holds too much. Beside it, a burndown against an ideal line, pace and estimated completion, and coverage broken down by how each case got into the run — requirement, suite, or hand-picked.

Two limits worth stating plainly. Time tracking is coarse: an optional six-bucket dropdown when you complete a case, not a timer and not a number, and there’s no report that aggregates it — don’t plan on effort analytics. And nothing is live: there are no websockets, so two testers in the same run won’t see each other’s results appear in real time. Outbound is the exception — webhooks fire on run created, completed, resumed and deleted, so real-time integration works in the direction that usually matters.