Our insurance-specific AI beats the market on accuracy and speed

Can AI actually understand insurance?

AI models are getting better at answering complex questions, but insurance presents a unique challenge. Answers are often buried across hundreds of pages of policy wordings, underwriting guidelines, endorsements and exclusions. Finding the right information is only part of the problem. Knowing when there isn’t enough information to answer is just as important.

We built General Magic’s Knowledge Retrieval (KR) pipeline specifically to solve this problem. To understand how much of a difference industry-specific AI makes, we benchmarked it against 7 leading frontier models and 2 Microsoft enterprise AI assistants on real Canadian home and auto underwriting questions. To keep the comparison fair, every system had the same setup and worked from the same fixed benchmark set of 202 documents, so each answer could be checked against a known source.

Here’s what we found:

Across multiple rounds of benchmarking, our KR pipeline answered Canadian home and auto insurance questions more accurately and faster than every leading AI model and enterprise assistant we tested.

In our latest round of testing, our system was 1.3x as accurate as the next-best system and 1.6x as accurate as the best frontier model, while answering roughly 10 seconds faster than the quickest alternative. On average, it responded 24 seconds faster than the other systems tested. In our first round of testing, it was over 4x more accurate than the average frontier model.

The hard part: finding the right information

The hard part of AI for insurance is not writing the answer. It is finding the right information across many long, detailed insurance documents: rules, exceptions and tables that link to one another. Indexing those documents and pulling back exactly what a question needs is where most systems fall down.

It is also hard because the answers depend on each other, and much of the information is not public. A single insurance question might require information from:

• Policy wordings

• Underwriting manuals

• Endorsements

• Eligibility guidelines

• Provincial regulations

• Carrier-specific bulletins

For example, determining whether a basement apartment is covered might require checking the main policy, the rental-use conditions and any applicable endorsements together. Miss one and the answer is incomplete. The example below shows the same pattern for a flooded basement.

Example underwriting question

My client in Ontario had water come into their basement after a heavy rainstorm flooded the street. Is that covered under their homeowners policy, or do they need overland water coverage?

PropertyOntarioHomeowners

1. Policy wordingCoverage conditions

   Loss caused by flood or surface water is not insured.

2. Underwriting guideEligibility rules

   Overland water is not available for properties in a high-risk flood zone.

3. EndorsementsExceptions

   Overland water coverage applies only if the endorsement is added.

Policy wording+Underwriting guide+Endorsements

One question. Multiple sources.

Not covered on the base policy. The client needs the overland water endorsement, and it isn’t available if the home is in a high-risk flood zone.

Illustrative example, not an actual benchmark result or coverage determination.

How we tested

We ran several rounds of extensive testing on Canadian personal lines insurance, with questions covering both home and auto. Each round used a large set of real underwriting questions, around 200 per round. Every item below was held the same across systems so the results compare like for like.

• Rounds of testing: Each round contained an average of 200 real underwriting questions. Every system answered the same questions within each round.

• Source material: Every system worked from the same fixed benchmark set of 202 documents, which it used both to retrieve information and to understand it. Much of the content was connected across documents, so answering a single question often meant looking at several documents together.

• Systems tested: Our full KR pipeline (our insurance-specific implementation of knowledge retrieval), seven leading frontier models from OpenAI and Anthropic, and, in the second round, two enterprise AI assistants from Microsoft: Microsoft Copilot and Copilot Chat (enterprise).

• Frontier model setup: We built a standard retrieval setup (often called naive RAG) so each model could search the same benchmark set as our pipeline. Every model received the same retrieved information, so any difference between them came from the model alone.

• Enterprise assistant setup: Each assistant used its own built-in document search over the same benchmark set. Although they share a name, Microsoft Copilot and Copilot Chat (enterprise) are separate products, so we tested and scored them separately.

• How answers were judged: Every answer was checked against a verified answer drawn from the source documents, so systems were scored on whether they found the right information. Underwriters validated those answers and reviewed the results, backed by real feedback from their day-to-day work. Each answer passed or failed, and in the second round answers were judged on substance, regardless of formatting or style.

• Declined answers: If a system declined to answer, it was marked wrong. No system could protect its score by staying quiet.

• Speed: Average time per successful answer, covering both the search and writing the answer.

How far ahead we scored

The KR pipeline scored far above every other system in both rounds of testing. In the first round it was over 4x more accurate than the average frontier model. In the latest round, with the enterprise assistants added, it was twice as accurate as the average of every other system tested.

In the latest round, it was 1.3x as accurate as Microsoft Copilot, the strongest alternative, 1.6x as accurate as the best frontier model and 3.7x as accurate as Copilot Chat (enterprise). The two Copilot products landed at opposite ends of the results, which is why we report them separately. In the first round, it was nearly 4x as accurate as the best frontier model.

The second round used updated grading that raised scores across the board, but our pipeline held steady in the low to mid 80s and kept a wide lead.

Faster, too

The KR pipeline was the fastest system in both rounds of testing, and on average answered about 24 seconds faster than the rest of the market.

The speed gap was consistent across rounds of testing. In the first round, the KR pipeline was also about 10 seconds faster than the next fastest system and about 30 seconds faster than the slowest frontier models. Getting the right material up front also significantly reduced what each correct answer costs to produce. The model reads less and reasons less, so the same design that makes the pipeline accurate and fast keeps it efficient too.

A raw score, on purpose

JEV is a classification model from TypeSafe AI, released in September 2026. It does not write text. You give it a question and a set of options, and it returns a choice or a score with a confidence level, in well under a second.

In a pipeline like ours, JEV could help in three places:

• Routing questions: sending each question to the right line of business, coverage or set of documents before the search begins.

• Ranking what comes back: replacing a traditional reranker by treating relevance as a classification problem. Rather than only asking which passage is most similar to the question, we can ask direct questions about each one: what it contains, whether it meets the question’s criteria and how useful it is. The strongest evidence then reaches the model first.

• Spotting missing answers: flagging when nothing retrieved actually answers the question, so the system can say so instead of guessing.

We chose not to use it here. We wanted a clear test of our pipeline on its own, with nothing else helping, so the results show how well it indexes documents and retrieves the right information. We also did not fine-tune or retrain any model: every model runs exactly as its provider ships it.

That makes these results a floor, not a ceiling. With JEV and other improvements to retrieval and classification, there is still a lot of room to raise these scores further.

Why specific wins

The results point to one conclusion: building AI specifically for insurance is what makes the difference, not picking a bigger model.

Swapping in a more powerful model barely helped. Given the same information, the best frontier model got only about half the answers right in the latest round, and the most expensive model scored among the lowest in both rounds. Systems that ran their own document search did better, and our architecture did best by a wide margin.

Insurance documents are hard for general tools. Answers are spread across sections, exceptions and tables, small differences in wording change the answer, and rules change often. General-purpose search treats an insurance document like any other file, which is why the more documents a question involved, the further ahead our architecture pulled.

Our architecture is built around lines of business, coverages, rules and exceptions, so it finds every piece an answer depends on before the model starts writing. That is also why it is faster: with the right material up front, the model reads less and reasons less to reach the right answer.

We have seen this pattern before. Coding went first, then law, and insurance is next. Harvey shows it in law: rent the model, then build the knowledge, workflows, citations and human checks around it. The model was never the advantage. Everything built around it was.

The AI labs build the engine. Specialists build the car.