On The Truthfulness of 'Surprisingly Likely' Responses of Large Language Models

TL;DR

This work introduces the concept of 'surprisingly likely' responses for large language models, inspired by crowd-based truthful information elicitation like the Bayesian Truth Serum. It defines a score $\tau(r,q) = \frac{P(r|q)}{P(r|'?')}$ to rank responses by their contextual likelihood relative to a prior, and evaluates this approach on TruthfulQA, COPA, and StoryCloze using open models without fine-tuning. Across multiple models, the method yields meaningful accuracy gains, particularly on TruthfulQA where improvements can reach up to $24$ percentage points, though some categories resist improvement and hybrid strategies may be needed. The results suggest a promising link between crowd-sourced truthfulness mechanisms and LLM training dynamics, with potential extensions in decoding, pre-training, or reinforcement learning to encourage more truthful behavior in practice.

Abstract

The principle of rewarding a crowd for surprisingly common answers has been used in the literature for designing a number of truthful information elicitation mechanisms. A related method has also been proposed in the literature for better aggregation of crowd wisdom. Drawing a comparison between crowd based collective intelligence systems and large language models, we define the notion of 'surprisingly likely' textual response of a large language model. This notion is inspired by the surprisingly common principle, but tailored for text in a language model. Using benchmarks such as TruthfulQA and openly available LLMs: GPT-2 and LLaMA-2, we show that the surprisingly likely textual responses of large language models are more accurate in many cases compared to standard baselines. For example, we observe up to 24 percentage points aggregate improvement on TruthfulQA and up to 70 percentage points improvement on individual categories of questions in this benchmark. We also provide further analysis of the results, including the cases when surprisingly likely responses are less or not more accurate.