Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction

TL;DR

This work tackles the challenge of evaluating and training large language models under weak supervision and potential deception. It introduces peer prediction as an incentive-compatible, ground-truth-free mechanism that rewards informative and honest answers via mutual predictability, and it formalizes both evaluation and training pipelines. The authors prove incentive compatibility and demonstrate, across models up to 405B parameters, that peer prediction can recover truthfulness lost to deceptive finetuning and outperform traditional LLM-as-a-Judge baselines, especially when expert–participant capability gaps are large. Empirically, the method differentiates stronger from weaker models, scales favorably with participant and expert pools, and exhibits an inverse scaling with the capability gap, making it practical for evaluating frontier models without strong supervision. These results advance scalable oversight by enabling reliable evaluation and training signals in settings where ground-truth labels are scarce or unreliable, with implications for safer and more trustworthy AI deployment.

Abstract

The evaluation and post-training of large language models (LLMs) rely on supervision, but strong supervision for difficult tasks is often unavailable, especially when evaluating frontier models. In such cases, models are demonstrated to exploit evaluations built on such imperfect supervision, leading to deceptive results. However, underutilized in LLM research, a wealth of mechanism design research focuses on game-theoretic incentive compatibility, i.e., eliciting honest and informative answers with weak supervision. Drawing from this literature, we introduce the peer prediction method for model evaluation and post-training. It rewards honest and informative answers over deceptive and uninformative ones, using a metric based on mutual predictability and without requiring ground truth labels. We demonstrate the method's effectiveness and resistance to deception, with both theoretical guarantees and empirical validation on models with up to 405B parameters. We show that training an 8B model with peer prediction-based reward recovers most of the drop in truthfulness due to prior malicious finetuning, even when the reward is produced by a 0.135B language model with no finetuning. On the evaluation front, in contrast to LLM-as-a-Judge which requires strong and trusted judges, we discover an inverse scaling property in peer prediction, where, surprisingly, resistance to deception is strengthened as the capability gap between the experts and participants widens, enabling reliable evaluation of strong models with weak supervision. In particular, LLM-as-a-Judge become worse than random guess when facing deceptive models 5-20x the judge's size, while peer prediction thrives when such gaps are large, including in cases with over 100x size difference.

Figures (14)

• Figure 1: Peer prediction-based truthfulness training improves ground-truth accuracy of deceptive models. Truthfulness training is performed with offline DPO on 120k paired answers with high vs low peer prediction score. Peer prediction with a 0.135B-parameter expert outperforms training on LLM-as-a-judge reward with either a 0.135B or a 7B judge.
• Figure 2: Peer prediction scores predict model honesty better than LLM-as-a-Judge scores do when the capability gap is large, and is therefore less susceptible to deception. Each curve shows honesty prediction loss on one given participant population by experts of varying sizes (0.135B-7B).
• Figure 3: The peer prediction pipeline. Peer prediction evaluates a participant (source) by measuring how much it helps the expert(s) predict the report of other participants (target). Experts are assumed to be honest but may be weak and easy to exploit. The obtained ranking of responses can be used for evaluation or for contrastive training.
• Figure 4: Larger ensembles tend to beat best-performing single experts, as predicted by Theorem \ref{['thm:crowd']}.(a) Ensemble improvement in honesty prediction (measured by honesty regression $R^2$) relative to the best expert in that ensemble, for ensembles of three or more experts. (b) Ensemble improvement for pairs of experts. (c) Ensemble improvement increases as ensemble size grows, when the aggregation exponent $\alpha\in[-1, -2]$. The ensemble output score is a weighted sum of the individual expert log probabilities, where each expert $i$ with size $s_i$ has weight $s_i^\alpha (\sum_j s_j^\alpha)^{-1}$.
• Figure 5: Deception resistance experiments on fully heterogeneous participants. (a)$\ldots$ where regression aims to tell apart all deceptive responses from all honest responses, regardless of which model generated them. (e)(f)(g)$\ldots$ where regression aims to tell apart responses of deceptive model X from those of honest model X, where X is Mistral 7B v0.3, Llama 3.1 8B, Gemma-2 9B respectively in the 3 subfigures. (b)(c)(d) Score distributions for peer prediction, LLM-as-a-Judge (6-shot), and LLM-as-a-Judge (0-shot) respectively, at various points in the performance curve. The discrete distributions of LLM-as-a-Judge scores are smoothed before visualization.

Theorems & Definitions (6)

• Theorem 1: Incentive Compatibility of Peer Prediction
• Theorem 2: Wisdom of the Crowd in Peer Prediction
• Remark 1
• Remark 2: Contributions in Proof Method
• Lemma 1: Data Processing Inequality
• Remark 3: Intuitive Interpretation of Assumption \ref{['ass:entropy']}