Building an Observatory Instead of Another Dashboard
A blank chart builder gets called flexibility, and it hands the reader a research project. Observatory starts from the question instead — and the presets, categories and role bundles all exist to keep it that way.
Every QA team builds a dashboard in the first month. Pass rate, defects by severity, tests by status, a pie chart of something. It looks great in the retrospective where it’s unveiled.
By month three nobody opens it.
The charts aren’t wrong. They’re accurate, they refresh, they render. The problem is that nobody can say what any of them is for — what decision changes if the donut shifts by six percent. They were built because the tool offered a chart builder and those were the fields available.
Why that happens, and how to design a dashboard people still open, is its own guide. This post is the other half: what we built instead. Observatory is a QA analytics surface where you pick a question — how fast are we automating our test suite? — and the chart is whatever answers it.
A blank chart builder is a question you’re being asked to answer
Hand someone an empty chart editor and you’ve handed them a research project. Pick a data source. Pick a dimension. Pick an aggregation. Pick a visualisation. Every one of those is a real decision, and getting all four right requires already knowing what you’re looking for.
So people pick what’s easy to plot rather than what’s worth knowing. Status distribution is easy. “Which of our automated tests hasn’t actually run in a build?” is not — and it’s the one that would change what you do this sprint.
Flexibility isn’t a feature if it costs the user the answer.
- How fast are we automating our test suite? Repositoryover timeline
- Which priority tests still rely on manual coverage? Repositoryby prioritystacked bar
- Are defects staying closed? Defectsstatus trendarea
Every curated insight carries its question as a first-class field alongside its title — “Are defects staying closed?”, “Which priority tests still rely on manual coverage?”, “Are all requirements backed by tests?” Picking one settles all four of those decisions for you, because they were never the interesting ones. They were the tax.
The custom chart builder still exists, and it should — occasionally you genuinely need something nobody anticipated. It’s just not the front door.
An empty heading teaches people to skim
The insights are grouped — Coverage, Automation, Execution, Quality, Defects, Requirements, Release Health, Team — and each group carries a one-line description of what it’s for rather than just a name. “Find gaps in protection.” “Gauge stability and flakiness.”
Empty categories aren’t rendered at all. A grid of eight headings where three have nothing under them trains people to skim past headings, and after that the grouping is decoration rather than navigation.
Collections go further and bundle whole panels by role, because a QA manager’s view isn’t the same six questions as a developer’s. They reference insights by id, and an id that no longer resolves is skipped rather than raised — renaming an insight should never be able to break somebody’s dashboard.
The number that catches automation lying
The scorecard is where question-first stops being an interface idea and starts paying off.
Alongside the automation percentage — the one every tool reports, computed from what your test cases claim — there are three more: how much automation is confirmed by build evidence, how much is unclaimed, and how much is stale.
Stale automation is the actionable one. A test case marked automated that no build has ever run is either automation that quietly broke, or automation that never existed. Either way it has been inflating your automation percentage, and the inflation compounds silently for as long as nobody looks.
You cannot get to that number from a chart builder, because you’d have to know to ask for it.
Numbers that agree with the pages they came from
One implementation decision does more for trust than anything visual.
The scorecard endpoint doesn’t compute its own coverage, execution and defect figures. It calls the same handlers the Coverage report, Execution Summary and defect-aging views use, and returns their raw output.
The reason is the failure it prevents. Independent implementations of “coverage” drift — not through carelessness but through ordinary maintenance, each one patched separately over a year. Then the scorecard says 71% and the coverage report says 68%, both are defensible, and every number on the screen loses authority at once. It’s the same argument that moved readiness scoring out of the browser after three copies produced two answers.
The endpoint also returns raw KPIs only — the 0–100 grading happens client-side. Scoring is presentation; the measurements are the product.
An insight that ends in a verb
Last piece. Charts get a generated narrative, and the shape of it is deliberate: a message, the reason, and an action.
“High priority test cases still lean heavily manual” — then the counts behind it, then what to do about it. Not a severity badge and a number. A finding with a next step.
Which is the whole thing the name is pointing at.
A dashboard shows you numbers. An observatory answers the question you actually walked in with.
Observatory, panels and insights
