Benchmark found 56% wrong answers on 55 documents

A head-to-head regulatory AI benchmark on 55 documents found a general-purpose model was wrong 56% of the time when asked to identify publication, effective and comment-close dates.

Archer compared its Evolv compliance system with a leading general-purpose large language model across six jurisdictions.

Results from the sample showed a wide gap. The general-purpose process produced correct answers on 44% of documents, returned wrong answers as valid in 25% of cases, and failed or timed out in 31%.

By contrast, Archer Evolv verified more than 95% of determinations outright. It routed the remaining 5% to an expert before use, and no wrong date reached production.

Confidence scores also failed as a safety check. Of the answers the general-purpose model rated as high confidence, 35% were still wrong.

Kayvan Alikhani on compliance risk

Kayvan Alikhani, Archer’s chief product and technology officer, said: “In compliance, an answer that is fast and cheap, but wrong, is worthless, and an answer you cannot trace is a liability.”

According to Archer, the test focused on a narrower question than general AI quality: how to make a high-stakes regulatory determination reliable, fast and affordable at scale.

In Overland Park, Kansas, Archer said its domain-focused process uses proprietary data sets and an expert-verified knowledge base for compliance work.

Speed was another part of the comparison. Per request, the general-purpose process averaged about four seconds per response within a five-second timeout.

Archer Evolv returned a verified date in roughly five-hundredths of a second, which the company said was about 80 times faster on repeat lookups.

In a 500-document corpus with 12 lookups a month for each document, a standard lookup pattern would create 6,000 determinations. Archer said its process cuts that to 500 because it computes once at ingestion and stores the verified result for later use.

As a result, Archer said the process avoids about 92% of inference calls in that example.

Alikhani said: “Archer’s purpose-built AI verified more than 95% of determinations in real time. That is the foundation that lets enterprises scale AI agents without losing control of the outcome.”

Archer said verified, source-traceable and expert-governed answers are needed before companies can safely deploy AI agents across an enterprise.

Archer said the full methodology, source data and case study are available on its website.