Oracle Reads Your Project, Not the Internet
Hawzu's AI answers from three separate sources — the product docs, your own specifications, and your test coverage — kept deliberately apart, each with exactly one way in.
Ask a general-purpose assistant whether your release is ready to ship and it will tell you that a 96% pass rate looks healthy and you should probably resolve the open criticals first.
That’s not wrong. It’s just the average opinion of everything ever written about QA, and your team may have written down that payments requires 100% on P1 tests and zero open blockers, full stop. The generic answer isn’t merely less useful than the specific one — on a sign-off screen it’s actively misleading, because it sounds informed.
The AI inside Hawzu is called Oracle, and almost all of the engineering in it is about what it reads rather than how it writes. What it can read, and what it must not confuse, turns out to be the entire product.
Because when something judges your release for you, the difference between what Hawzu’s own documentation says, what your team wrote down, and what your tests already cover is the difference between an informed answer and a plausible one.
Three sources that must never merge
There are three separate bodies of knowledge, and they answer three different questions.
The product documentation answers how does Hawzu work. It’s the same for every customer, so there is one shared copy.
Canon is your own material — specs, acceptance criteria, standards, the release policy someone wrote in 2024. It answers what did we promise. It belongs to a project, and it is the most sensitive data in the system.
The coverage index holds an entry for every test case in the project and answers is this behaviour already tested.
Keeping the last two apart is the decision that gets questioned most, because both describe your product and both are stored the same way. The rule is short: evidence comes from your documents, coverage comes from your repository. A documented behaviour doesn’t become less documented because a test happens to cover it, and an undocumented one doesn’t become documented because an existing test looks similar.
Merge them and the system can no longer tell you the thing you most need to know: that a behaviour is specified and untested.
One way in for each
Every source has exactly one way to read it, and that is enforced rather than encouraged.
The reason is a specific, ugly failure. The documentation search doesn’t filter by customer at all — which is correct, because the docs are the same for everyone and there’s nothing to filter by. If Canon searches were allowed to reuse it, or even to resemble it closely enough that someone copied it, that missing filter would put one customer’s documents into another customer’s answers.
The same edge exists inside a single customer. Canon belongs to a project, and a question asked in one project reads that project’s documents and never another’s. Getting that wrong leaks one project’s specifications into another’s answers, which is the same kind of bug as the missing filter wearing better clothes.
Three outcomes, not two
Here’s the smallest decision on the list and the one I’d defend hardest.
When Oracle checks whether a behaviour is already tested, it can come back with three things: matches, “the index is unavailable”, or “this project has nothing indexed yet”. Not two.
Collapse the last two into “nothing matched” — the natural shape, the shape a plain yes-or-no gives you — and the feature inverts. The index is briefly down, so nothing comes back, so nothing looks covered, so the generator confidently proposes tests for behaviours you already test thoroughly. A failure behind the scenes silently becomes a strong, wrong claim about your tests.
An error that reads as an answer is worse than an error. Any search feature needs to tell I looked and found nothing apart from I could not look.
There’s a cost rule in the same place that’s load-bearing rather than tidy: before indexing a test case again, Hawzu checks whether its text actually changed. Every test-case save reaches the index, and every indexing call is a paid request, so unchanged text must not cost one. That’s not an optimisation you add later — without it, a bulk edit of 5,000 cases is a surprise invoice.
The bug that proved the point
The best argument for all this is a bug we shipped.
The release summary looks up the Canon material relevant to the release being judged, so the model can assess it against your documented criteria rather than its own general knowledge. It built that lookup from the release title — but it looked for the title under one name, while the screen had always sent it under another.
The title was always empty. Every summary had been searching with generic words alone. The feature ran, returned plausible summaries, and nobody noticed, because an answer that sounds well-sourced and an answer that is well-sourced read identically from outside.
Two things changed. The summary now reads the same release analysis the Release Analysis screen reads, rather than narrating numbers the browser put together and sent — so the model can state no number Hawzu didn’t measure. And it returns its Canon citations even when the list is empty, so the screen can tell “assessed against your documented criteria” apart from “assessed against a model’s general knowledge.”
That empty list is the whole philosophy in one detail. An AI feature that can’t tell you what it read is asking to be trusted. One that shows you the documents — including when there weren’t any — is asking to be checked.
An answer you can’t trace isn’t informed. It’s just confident.
How answers cite your documents
