The chatbot was confident and wrong

There is a chatbot inside the Healing Oasis admin portal. The first version of it hallucinated.

Hallucination in a business tool is almost never spectacular. It is a confident, well-formatted, entirely plausible answer handed to somebody who has no particular reason to doubt it. That is worse than an error. An error stops you.

Why it happened

The first build answered from what it had been given up front. That is fine for the things that stay true — what a programme covers, how certification works, what the school is. It is not fine for anything operational, because operational facts change after you load them and a snapshot has no way of knowing it has gone stale.

Enrolment is a state. States move. An answer assembled from a snapshot of a state is a guess with good grammar.

The rebuild

The second version is agentic. Every question triggers a call for current data — it does not answer from what it remembers the data being. And it refuses outside its grounding: if the tools do not come back with something that answers the question, it says so.

The refusal is the feature, not the compromise. A tool that says “I don’t have that” gets trusted with the next question. A tool that guesses once, convincingly, never does again.

How I check what a model produces

The same discipline applies to the code, not only to the chatbot.

• Sandboxes first — Shopify development stores and Stripe test mode — so nothing touches production or real money.
• Deterministic scripts for the security-sensitive paths, because that is exactly where a model’s confidence is worth least.
• Evals on anything with a model in the loop.
• Nothing reaches a client without me reading it.

Only the last one is really a policy. The rest are tools, and tools get swapped. The accountability does not move.

One claim I am not going to round up

I keep a list of claims about my own work in a repo, so that anything I say out loud has somewhere to be checked. Sixty-eight lines. Two of them are false, and one of the two is directly about this:

[x] I've fixed a hallucination problem with grounding and guardrails
[x] I've built an agent that reads live business data, not a static corpus
[ ] I've written evals that gate a deploy

I run evals. They do not gate a deploy. Those are two different sentences and the gap between them is a CI step I have not built. The chatbot already has the corpus and the failure history to build a set from, so it is a cheap gap to close — but it is open today, and “we run evals that block a bad deploy” would be a nicer thing to say than a true one.

I would keep the failure anyway. A chatbot that hallucinated is a bad afternoon. A chatbot that hallucinated, got diagnosed, and got rebuilt so that every answer has to come back from a fetch is the only kind of evidence that you are still paying attention.