Oracle Reads Your Project, Not the Internet
Hawzu's AI is grounded in three separate corpora — the product docs, your own specifications, and your test coverage — kept deliberately apart, each behind exactly one read path.
Ask a general-purpose assistant whether your release is ready to ship and it will tell you that a 96% pass rate looks healthy and you should probably resolve the open criticals first.
That’s not wrong. It’s just the average opinion of everything ever written about QA, and your team may have written down that payments requires 100% on P1 tests and zero open blockers, full stop. The generic answer isn’t merely less useful than the specific one — on a sign-off screen it’s actively misleading, because it sounds informed.
The AI inside Hawzu is called Oracle, and almost all of the engineering in it is retrieval rather than generation. What it can read, and what it must not confuse, turns out to be the entire product.
Because when something judges your release for you, the difference between what Hawzu’s own documentation says, what your team wrote down, and what your tests already cover is the difference between an informed answer and a plausible one.
Three corpora that must never merge
There are three separate bodies of knowledge, and they answer three different questions.
The product documentation answers how does Hawzu work. It’s the same for every customer, so it’s a single global corpus.
Canon is your own material — specs, acceptance criteria, standards, the release policy someone wrote in 2024. It answers what did we promise. It is per-customer, per-project or per-workspace, and it is the most sensitive data in the system.
The coverage index holds one vector per test case and answers is this behaviour already tested.
Keeping the last two apart is the decision that gets questioned most, because both describe your product and both live in the same vector database. The code states the separation as a formula: evidence comes from your documents, coverage comes from your repository. A documented behaviour doesn’t become less documented because a test happens to cover it, and an undocumented one doesn’t become documented because an existing test looks similar.
Merge them and the system can no longer tell you the thing you most need to know: that a behaviour is specified and untested.
One read path each
Every corpus has exactly one function that reads it, and that is enforced rather than encouraged.
The reason is a specific, ugly failure. The documentation search function takes no tenant filter at all — which is correct, because the docs corpus is global and there’s nothing to filter by. If Canon retrieval were allowed to reuse that function, or even to resemble it closely enough that someone copied it, the same missing filter becomes a cross-customer data leak.
Scope inside Canon has a related sharp edge. A project-scoped question returns project documents plus workspace-level ones, because a workspace standard applies to every project under it. A workspace-scoped question returns only workspace documents. Getting that backwards leaks one project’s specifications into another’s answers, which is the same class of bug as the missing filter wearing better clothes.
Three outcomes, not two
Here’s the smallest decision on the list and the one I’d defend hardest.
When the coverage index is asked whether a behaviour is already tested, it can return three things: results, “the index is unavailable”, or “this project has nothing indexed yet”. Not two.
Collapse the last two into “nothing matched” — the natural shape, the shape a boolean gives you — and the feature inverts. The vector database is briefly down, so no candidates come back, so nothing looks covered, so the generator confidently proposes tests for behaviours you already test thoroughly. The failure of a dependency silently becomes a strong, wrong claim about your test suite.
An error that reads as an answer is worse than an error. Any retrieval feature needs to distinguish I looked and found nothing from I could not look.
There’s a cost discipline in the same layer that’s load-bearing rather than tidy: indexing checks a content hash before embedding. Every test-case write reaches the indexer and every embedding is a paid request, so unchanged text must not spend one. That’s not an optimisation you add later — without it, a bulk edit of 5,000 cases is a surprise invoice.
The grounding bug that proved the point
The best argument for all this is a bug we shipped.
The release summary endpoint retrieves Canon material relevant to the release being judged, so the model can assess it against your documented criteria rather than its own priors. It built that retrieval query from the release title — read from one field name, while the client had always sent another.
The title was always empty. Every call had been retrieving against the generic vocabulary alone. The feature ran, returned plausible summaries, and nobody noticed, because a grounded-sounding answer and a grounded answer read identically from outside.
Two things changed. The endpoint now reads the same computed release analysis the Release Analysis screen reads, rather than narrating a metrics blob the browser assembled and posted — so the model can state no number the server didn’t measure. And it returns its Canon citations even when the list is empty, so the interface can distinguish “assessed against your documented criteria” from “assessed against a model’s general priors.”
That empty list is the whole philosophy in one field. An AI feature that can’t tell you what it read is asking to be trusted. One that shows you the documents — including when there weren’t any — is asking to be checked.
An answer you can’t trace isn’t grounded. It’s just confident.
How answers are grounded and cited
