QA Dashboards: Building One People Still Open in Month Three

A QA dashboard is a standing answer to a small number of recurring questions. Not a display of available metrics — a set of answers to things somebody asks often enough that looking them up by hand has become annoying.

That definition rules out most QA dashboards, which is the point. The typical one is built in an afternoon from whatever the tool offered, looks impressive in the retro where it’s unveiled, and is dead by the third month. Not because anything broke — because nobody could say what any panel was supposed to change.

Why dashboards die

Three causes, in roughly this order.

They were built from fields, not questions

An empty chart builder asks four questions before you get anything: which data source, which dimension, which aggregation, which visualisation. Answering all four correctly requires already knowing what you’re looking for. So people plot what’s easy — status distribution, test count by priority — rather than what’s worth knowing, and end up with charts that are accurate and inert. The inversion — choosing a question and letting the chart follow from it — is what a builder-first tool asks you to do for yourself.

They show state without movement

A pie chart of current test statuses tells you where things stand and nothing about where they’re going. Almost every QA question that matters is about direction — is coverage improving, is the defect backlog growing, are we automating faster than we’re writing — and direction needs a time axis.

Nobody owns them

A dashboard with no owner drifts. A filter that was right in March is wrong in September; a chart pointed at a folder that’s since been reorganised now shows nothing and nobody notices. Six months on, people have quietly learned not to trust it, which is worse than not having it.

Design from the question backwards

The useful exercise takes ten minutes: write down the questions your team actually asks in standups, release meetings and retros. Then keep only the ones where the answer would change what somebody does.

You’ll usually end up with five to eight, and they tend to look like:

Anything that doesn’t earn a place on that list can be a report you run on demand. Dashboards should be short: a panel nobody looks at costs attention from the ones they do.

Charts that carry information, and ones that don’t

Trends over snapshots. A line over eight weeks beats a number today for almost every QA metric. The number tells you where you are; the line tells you whether what you’re doing is working.

Rates with their denominators. Pass rate without “how much has been executed” is a biased sample early in a cycle. If a panel shows a percentage, it should show the base.

Distribution over average. Average defect age hides the one ticket that’s been open for 400 days. Show the distribution, or at least the oldest.

Absolute counts, carefully. “142 test cases” isn’t information unless something compares it — to last month, to a target, to another area.

Keep the numbers agreeing with each other

The fastest way to kill trust in a dashboard is for it to disagree with a report. If the scorecard says coverage is 71% and the coverage report says 68%, both are now suspect, and every other number on the screen loses authority with them.

This happens through ordinary maintenance rather than carelessness: two independent implementations of “coverage” get patched separately over a year and drift apart. The fix is architectural — the dashboard should read the same computation the report does, not its own copy of the logic.

How this works in Hawzu

Observatory inverts the blank-builder problem: you choose a question, and the chart is what the question resolves to. Every curated insight carries its question as a first-class field — “How fast are we automating our test suite?”, “Which priority tests still rely on manual coverage?”, “Are defects staying closed?” — and picking one selects the data source, aggregation and visualisation for you, because those were never the interesting decisions.

Insights are grouped into categories that each carry a one-line purpose rather than just a name (“Find gaps in protection”, “Gauge stability and flakiness”), and empty categories aren’t rendered — a grid of headings where three are blank teaches people to skim past headings.

Collections go a step further: a named bundle of insights with an audience attached, added to a panel in one click, so a QA manager’s view isn’t assembled one chart at a time. They reference insights by id, and an id that no longer resolves is skipped rather than raised — renaming an insight can’t break somebody’s dashboard; the worst case is one card fewer. The custom chart builder still exists for the things nobody anticipated. It just isn’t the front door.

The Project Health scorecard grades automation, coverage, execution, quality and defect health, and it’s built to stay consistent with the rest of the product: the endpoint doesn’t compute its own figures, it calls the same handlers the Coverage report, Execution Summary and defect-aging views use, and returns their raw output. Scoring happens client-side, because scoring is presentation and the measurements are the product.

The metric worth knowing about is stale automation — cases marked automated that no build has ever run. It sits beside the automation percentage your test cases claim, and the gap between the two is the finding: automation that quietly broke, or automation that never existed. It has been inflating your coverage number for as long as nobody looked, and you can’t ask a chart builder for it, because you’d have to know to ask.

Charts also carry a generated narrative shaped as a message, the reason behind it, and an action — “High priority test cases still lean heavily manual”, then the counts, then what to do about it. That’s the difference between a dashboard and an observatory: one displays, the other is pointed at something.