Large language models say the darnedest things. As much as large language models (LLMs for short) like ChatGPT, Claude, or Bard have amazed the world with their ability to answer a whole host of questions, they’ve also shown a disturbing propensity to spit out information created out of whole cloth. They’ve falsely accused someone of seditious conspiracy, leading to a lawsuit. They’ve made up facts or fake scientific studies. These are hallucinations, a term that has generated so much interest that it was declared Dictionary.com’s word of 2023.

LLMs’ tendency to make stuff up may be the single biggest factor in holding the tech back from broader adoption. And for the many thousands of companies that have built their own products off LLMs like ChatGPT, the idea that these systems “confabulate” presents a major legal and reputational risk. No surprise, then, that there’s now a wave of companies racing to help companies minimize the damage of hallucinations.

In November, in an attempt to quantify the problem, Vectara, a startup that launched in 2022, released the LLM Hallucination Leaderboard. The range was staggering. The most accurate LLMs were GPT-4 and GPT-4 Turbo, which Vectara found hallucinate 3% of the time when asked to summarize a paragraph of text. The worst performer was Google’s PALM 2 Chat, which had a 27% hallucination rate.

Nick Turley, the product lead of ChatGPT, which last year became the fastest-growing consumer app in history, says OpenAI has been making strong incremental progress on cutting down hallucinations. The latest versions of ChatGPT, for example, are now more open about what it doesn’t know and refuses to answer more questions. Still, the issue may be fundamental to how LLMs operate. “The way I’m running ChatGPT is from the perspective that hallucinations will be a constraint for a while on the core model side,” says Turley, “but that we can do a lot on the product layer to mitigate the issue.”

Measuring hallucinations is tricky. Vectara’s hallucination index isn’t definitive; another ranking from the startup Galileo uses a different methodology, but also finds ChatGPT-4 has the fewest hallucinations. LLMs are powerful tools, but they’re ultimately grounded in prediction—they use probabilistic calculations to predict the next word, phrase, or paragraph after a given prompt. Unlike a traditional piece of software, which always does what you tell it, LLMs are what industry experts refer to as “non-deterministic.” They’re guessing machines, not answering machines.

LLMs don’t reason on their own and they can have trouble distinguishing between high- and low-quality sources of information. Because they’re trained on a wide swath of the internet, they’re often steeped in a whole lot of garbage information. (You’d hallucinate too if you read the entire internet.)

To measure hallucinations, Vectara asked LLMs to perform a very narrow task: summarize a news story. It then examined how often the systems invented facts in their summaries. This isn’t a perfect measurement for every LLM use case, but Vectara believes it gives an approximation of how they can take in information and reliably reformat it.

“The first step to awareness is quantification,” says Amin Ahmad, Vectara’s chief technology officer and cofounder, who spent years working at Google on language understanding and deep neural networks.

There are two main schools of thought when it comes to hallucination mitigation. First, you can try to fine-tune your model, but that’s often expensive and time-consuming. The more common technique is called Retrieval Augmented Generation (RAG), and Vectara is one of the many companies that now offers a version of this to their clients.

In a very simplified sense, RAG works like a fact-checker for AI. It will compare an LLM’s answer to a question with your company’s data—or, say, to your internal policies or a set of facts. The combined LLM and RAG system will then tweak the LLM’s answer to make sure it conforms to this given set of limitations.

Vectara has seen a big demand from companies who need help building a chatbot or other question-and-answer style systems, but can’t spend months or millions tweaking its own model. The first wave of Vectara’s customers was heavily concentrated in customer support and sales, areas where, in theory, a 3% error rate may be acceptable.

In other industries, that kind of hallucination rate could mean life and death. Ahmad says Vectara has seen growing interest from the legal and biomedical fields. It’s easy to imagine a chatbot that eventually revolutionizes these fields, but imagine a lawyer or doctor that makes up facts 3 to 27% of the time.