Insurance calls are long, regulated and full of real actions, so an AI agent that hallucinates can do real harm, like telling a caller their claim is filed when it isn’t. The usual defense is post-call detection, and we used it too. But detection only works after the caller has hung up, so we asked a different question: instead of catching hallucinations faster, how do you remove the places they can happen? This article shares how we redesigned our agents around that question, what we found, and where the limits still are.
Liberate Research Team
Hallucinations are a hard problem anytime a language model talks to a customer. In insurance, the problem gets even harder. Insurance calls are long. FNOL calls can easily run 20 minutes with 30+ turns, and every fact the caller gives in the first minute has to be accurate by the end of the call.
Insurance calls don’t just answer simple questions. They have to do real things like pulling up a policy or filing a claim. If the model that does the talking is also the one deciding when to act, it can tell a caller "I've filed your claim" without ever having filed it. On top of that, the language is regulated. Some disclosures have to be read word for word at a specific moment, and some types of losses have to go straight to a person. A model treats those rules like any other instruction, which means it follows them most of the time, not every time. Hallucinating the wrong name to a customer is annoying and erodes trust. But hallucinating that a claim was filed can leave a customer standing in a flooded kitchen thinking help is on the way when nothing happened at all.
The standard defense to LLM hallucinations is detection. Run the conversation past a second model after the call, flag the discrepancies, then alert an engineer. We did this too. Our TruthGuard evaluations in our first-generation agents compared what the agent said against what it actually did, and they caught real problems, like the agent confirming a claim filing that never fired. But detection has a structural flaw. It happens after the damage. By the time the eval runs, the caller has hung up believing something false.
So we flipped the question. Instead of asking how to catch hallucinations faster, we asked how to remove the places where a hallucination can happen at all. We think of it as a surface area problem. Every point where a language model's output directly drives an action, a fact, or a required phrase is a point where the system can be wrong. Our current architecture exists to make that surface as small as we can get it.
What we built
Our current agents are managed by an orchestrator, which uses a deterministic map of the task at hand. For a job like filing an auto claim, the map defines the milestones the conversation needs to reach, such as authenticating the caller, capturing the loss details, and creating the draft claim. At each milestone there are slots, the specific facts we need, like first name, last name, and loss type. We call the pattern milestone slot filling, and it changes where the intelligence in the system actually sits.
On every turn, when the caller says something, three narrow model calls run in parallel. One extracts facts. One assesses intent, checking whether the caller is trying to do something different. One reads sentiment. Each has its own small system prompt and its own narrow job. Their outputs feed the orchestrator, which is not a model. It's business logic that compares the current state against the map and decides what happens next.
That structure moves the dangerous work out of the model entirely.
Tool calls are deterministic. When the conversation reaches a checkpoint that touches a carrier's System of Record, reading policy data or writing a draft claim, the orchestrator fires the call. The fields passed in are the predefined slots, not whatever a model decides is probably the right information. We know exactly what goes in and exactly what comes back. The agent can't claim it filed a claim it didn't file, because saying so and doing so are no longer the same system.
Memory lives in state, not in the transcript. The context window holds only the last 10 messages. Everything the caller has told us is stored as structured facts outside the model and pushed back into context only when needed. The caller's name doesn't need to sit in the window for 20 minutes waiting to be forgotten. It sits in state, correct, until it's used.
Escalation is a business rule. There are scenarios, like certain loss types or loss locations, that a given carrier doesn't want automated. If the captured loss type is one we don't handle, the orchestrator short-circuits the conversation and initiates a warm transfer to a representative immediately. The model's only job in that chain is extracting the loss type correctly. Everything after is code.
What we found
Most hallucination risk disappears when the model stops owning actions. The phantom-action failure, the most dangerous class we saw in first-generation agents, is structurally impossible when tool calls fire from the orchestrator with fixed fields. We didn't make the model more truthful about what it did. We removed its ability to be the one reporting.
A bounded context window is a quality control and a security control at once. Holding 10 messages instead of a full transcript means inference runs faster and recall stops degrading over the length of the call, since facts live in state rather than fading into the middle of a long window. It also breaks the layering that prompt injection depends on. An attack built up across many turns gets evicted before it matures.
Structure made onboarding faster, not slower. We expected a map-based design to demand heavy pre-planning compared to writing a prompt. The opposite happened. The old system prompt was an unstructured document with no management tooling. The map gives every new carrier a standard starting flow, our point of view on how an auto claim call should run, and gets them most of the way there before modifications for their regulatory and business requirements.
The real limit
The surface isn't zero and we don't claim it is. The model still works at the edges. Fact extraction can mishear a loss type, intent assessment can misread a pivot, and a generated response can phrase something imperfectly. That's why confirmation of important facts is built into the flow and why escalation to a person is a designed outcome rather than a failure state.
The tradeoff has a cost too. A deterministic map is more system to build and maintain than one long prompt, and the underlying pieces, like the graph framework and the voice runtime, are available to anyone. What isn't available is the map itself, the accumulated knowledge of how a claims conversation should run, which conditions need an escape hatch, and what a carrier in a given state requires. That knowledge is the work, and it's where we've chosen to spend our effort.
The call this is built for
The reason any of this matters is a person twenty minutes into reporting the worst day of their year. They've given their name, their policy number, and a description of the accident, and what they need now is for the system to have actually heard all of it, to file what it says it filed, and to hand them to a person the moment it can't help. A hallucination in that moment isn't a benchmark miss. It's a broken promise at exactly the moment the promise was the product.
We think the agents that earn a place in regulated industries won't be the ones with the best post-hoc detection. They'll be the ones designed so there's less and less to detect. Hallucination isn't a model behavior to correct. It's a surface to shrink, and every piece of the conversation that moves from probability to code shrinks it.
If you're working on the same problem and draw the line between model and code somewhere different, we'd like to compare notes.
Key takeaways
Post-call detection has a structural flaw: by the time it runs, the caller has already hung up believing something false.
When tool calls fire from the orchestrator with predefined fields, the agent can’t claim to have filed a claim it never filed. Saying and doing are no longer the same system.
Keeping only the last 10 messages in context, with caller facts stored in structured state, keeps recall steady across a 20-minute call and helps disrupt prompt injection attacks built up over many turns.
Escalation is a business rule, not a model judgment. If a loss type falls outside what a carrier wants automated, the orchestrator triggers a warm transfer to a representative right away.
A map-based design made carrier onboarding faster, not slower, by giving each carrier a standard starting flow to adjust for its own regulatory and business requirements.
By clicking “Accept”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.
