The Consensus Game: Language Model Generation via Equilibrium Search

TL;DR

This work reframes language-model decoding as a training-free, two-player imperfect-information signaling game between a generator and a discriminator, aiming to reconcile generative and discriminative queries. It introduces equilibrium-ranking, a no-regret, KL-regularized approach (piKL) that finds equilibria reflecting a consensus about correctness, anchored by initial LM priors. Across six QA benchmarks, equilibrium-ranking yields substantial improvements over standard decoding strategies and even enables smaller models to rival larger ones, underscoring the potential of game-theoretic tools to enhance truthfulness and consistency in LMs. The method is compatible with existing prompting and reasoning techniques and offers a computationally efficient, training-free pathway to more reliable LM predictions.

Abstract

When applied to question answering and other text generation tasks, language models (LMs) may be queried generatively (by sampling answers from their output distribution) or discriminatively (by using them to score or rank a set of candidate outputs). These procedures sometimes yield very different predictions. How do we reconcile mutually incompatible scoring procedures to obtain coherent LM predictions? We introduce a new, a training-free, game-theoretic procedure for language model decoding. Our approach casts language model decoding as a regularized imperfect-information sequential signaling game - which we term the CONSENSUS GAME - in which a GENERATOR seeks to communicate an abstract correctness parameter using natural language sentences to a DISCRIMINATOR. We develop computational procedures for finding approximate equilibria of this game, resulting in a decoding algorithm we call EQUILIBRIUM-RANKING. Applied to a large number of tasks (including reading comprehension, commonsense reasoning, mathematical problem-solving, and dialog), EQUILIBRIUM-RANKING consistently, and sometimes substantially, improves performance over existing LM decoding procedures - on multiple benchmarks, we observe that applying EQUILIBRIUM-RANKING to LLaMA-7B outperforms the much larger LLaMA-65B and PaLM-540B models. These results highlight the promise of game-theoretic tools for addressing fundamental challenges of truthfulness and consistency in LMs.

Figures (1)

• Figure 1: (Left) Overview of our approach. Differing LM queries fail to exhibit consensus about the answer to a factual question. By reconciling predictions between generative and discriminative LM queries using the consensus game, we obtain an accurate prediction. (Right) Structure of the consensus game, a two-player sequential signaling game with imperfect information. First, the environment (N) uniformly samples a correctness parameter. A generator (G) conditioned on this parameter produces a natural language string from a set of candidates. The discriminator (D) only observes this string and must predict the correctness parameter sampled by environment. If the discriminator correctly identifies this parameter, then both players receive a reward of 1. The dashed line connects nodes that are indistinguishable by the discriminator, since the discriminator does not observe the correctness parameter. By computing regularized equilibrium strategies for this game, we obtain predictions that reflect a consensus between the generator and discriminator.

Theorems & Definitions (3)

• Proposition 1: bakhtin2022mastering
• Proposition 2: Folklore connection between regret minimization and equilibria
• Proposition 3: Jacob22:Modeling