Incentivizing Truthful Language Models via Peer Elicitation Games

TL;DR

This paper tackles the problem of aligning large language models to be truthful and consistent without supervision or fine-tuning. It introduces Peer Elicitation Games (PEG), a training-free, multi-agent framework in which a generator and multiple discriminators engage in peer evaluation, with utilities derived from determinant-based mutual information to incentivize truthful reporting. The authors prove dominant truthfulness, sublinear no-regret dynamics via online mirror descent, and last-iterate convergence to a truthful Nash equilibrium. Empirically, PEG yields substantial factual accuracy gains across ARC, MMLU, GPQA, and demonstrates that smaller models can rival larger ones when coordinated through PEG. This approach offers a scalable, supervision-free path to more trustworthy LLMs and opens avenues for deployment in resource-constrained settings.

Abstract

Large Language Models (LLMs) have demonstrated strong generative capabilities but remain prone to inconsistencies and hallucinations. We introduce Peer Elicitation Games (PEG), a training-free, game-theoretic framework for aligning LLMs through a peer elicitation mechanism involving a generator and multiple discriminators instantiated from distinct base models. Discriminators interact in a peer evaluation setting, where utilities are computed using a determinant-based mutual information score that provably incentivizes truthful reporting without requiring ground-truth labels. We establish theoretical guarantees showing that each agent, via online learning, achieves sublinear regret in the sense their cumulative performance approaches that of the best fixed truthful strategy in hindsight. Moreover, we prove last-iterate convergence to a truthful Nash equilibrium, ensuring that the actual policies used by agents converge to stable and truthful behavior over time. Empirical evaluations across multiple benchmarks demonstrate significant improvements in factual accuracy. These results position PEG as a practical approach for eliciting truthful behavior from LLMs without supervision or fine-tuning.

Figures (6)

• Figure 1: Comparison of the consensus game and PEG: when multiple discriminator LLMs independently evaluate the generator's output and are rewarded based on mutual agreement, their collective judgment aligns more closely with true answers.
• Figure 2: Overview of our method: multiple discriminators independently evaluate the response provided by the generator, while each discriminator is rewarded based on mutual agreement with peers via PEG. This setup incentivizes truthful reporting for discriminators and aligns the generator without requiring ground-truth labels.
• Figure 3: Accuracy comparison between original model outputs (D) and PEG majority vote answers.
• Figure 4: An illustrative example of PEG's peer evaluation process: (1) The generator answers a list of questions. (2) Discriminators evaluate these answers, with some providing untruthful reports. (3) Determinant-based utilities penalize non-truthful discriminators, incentivizing them to align their future reports with the ground truth.
• Figure 5: Different Batch Size Effect on Majority Vote Accuracy

Theorems & Definitions (27)

• Lemma 1
• Theorem 1
• Theorem 2: Last-iterate Convergence to Nash
• Definition 1: Strategy
• Definition 2: Informative Peer
• Definition 3
• Lemma 2: Strict Information Monotonicity
• Lemma 3
• proof
• Definition 4: Bregman divergence