Major research into ‘hallucinating’ generative models advances reliability of AI

Researchers from the Department of Computer Science have made a significant advance towards ensuring that information produced by generative artificial intelligence (AI) is robust and reliable. In a new study published in Nature, they demonstrate a novel method to detect when a Large Language Model (LLM) is likely to ‘hallucinate’ (i.e. invent facts that sound plausible but are imaginary). This advance could open up new ways to deploy LLMs in situations where ‘careless errors’ are costly such as legal or medical question-answering. 

The researchers focused on hallucinations where LLMs give different answers each time they are asked a question － even if the wording is identical － known as ‘confabulating’. 

LLMs are highly capable of saying the same thing in many different ways, which can make it difficult to tell when they are certain about an answer and when they are literally just making something up. With previous approaches, it wasn’t possible to tell the difference between a model being uncertain about what to say versus being uncertain about how to say it. But our new method overcomes this. Study co-author Dr Sebastian Farquhar

To do this, the research team developed a method grounded in statistics and using methods that estimate uncertainty based on the amount of variation (measured as entropy) between multiple outputs. Their approach computes uncertainty at the level of meaning rather than sequences of words, i.e. it spots when LLMs are uncertain about the actual meaning of an answer, not just the phrasing. To do this, the probabilities produced by the LLMs, which state how likely each word is to be next in a sentence, are translated into probabilities over meanings. 

The new method proved much better at spotting when a question was likely to be answered incorrectly than all previous methods, when tested against six open-source LLMs (including GPT-4 and LLaMA 2). This was the case for a wide range of different datasets including answering questions drawn from Google searches, technical biomedical questions, and mathematical word problems. The researchers even demonstrated how semantic entropy can identify specific claims in short biographies generated by ChatGPT that are likely to be incorrect. 

Our method basically estimates probabilities in meaning-space, or 'semantic probabilities'. The appeal of this approach is that it uses the LLMs themselves to do this conversion. Study co-author Jannik Kossen

By detecting when a prompt is likely to produce a confabulation, the new method can help make users of generative AI aware when the answers to a question are probably unreliable, and to allow systems built on LLMs to avoid answering questions likely to cause confabulations. A key advantage to the technique is that it works across datasets and tasks without a priori knowledge, requiring no task-specific data, and robustly generalises to new tasks not seen before. Although it can make the process several times more computationally costly than just using a generative model directly, this is clearly justified when accuracy is paramount. 

Currently, hallucinations are a critical factor holding back wider adoption of LLMs like ChatGPT or Gemini. Besides making LLMs unreliable, for example by presenting inaccuracies in news articles and fabricating legal precedents, they can even be dangerous, for example when used in medical diagnosis. 

Getting answers from LLMs is cheap, but reliability is the biggest bottleneck. In situations where reliability matters, computing semantic uncertainty is a small price to pay. Senior study author Professor Yarin Gal

The study, Detecting Hallucinations in Large Language Models Using Semantic Entropy, was partially funded by the Alan Turing Institute. Dr Sebastian Farquhar, Jannik Kossen, and Lorenz Kuhn share lead authorship of the paper. 

Professor Gal’s research group, the Oxford Applied and Theoretical Machine Learning Group, is home to this and other work pushing the frontiers of robust and reliable generative models. Building on this expertise, Gal now acts as Director of Research at the UK’s AI Safety Institute. “There is an urgent need for research that measures and mitigates emerging risks from generative AI,” he says. 

The researchers highlight that confabulation is just one type of error that LLMs can make. ‘Semantic uncertainty helps with specific reliability problems, but this is only part of the story,’ explained Dr Farquhar. ‘If an LLM makes consistent mistakes, this new method won’t catch that. The most dangerous failures of AI come when a system does something bad but is confident and systematic. There is still a lot of work to do.’ 

View the full study in Nature at https://www.nature.com/articles/s41586-024-07421-0