Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating Fair Consensus Statements with Social Choice on Token-Level MDPs

Carter Blair, Kate Larson

TL;DR

This work addresses the challenge of provably fair consensus statement generation by modeling open-ended text formation as a token-level MDP. Each agent contributes a policy that defines token-level rewards, enabling fairness notions from social choice theory to be directly integrated into generation. The authors develop two main contributions: (i) a stochastic policy $\pi^*$ that optimizes Nash Welfare to produce an ex-ante core lottery over complete statements, and (ii) deterministic, egalitarian-welfare-based search methods (finite lookahead and beam search) for producing a single consensus statement. Empirically, the approach yields improved worst-case agent alignment (lower Egalitarian Perplexity) compared with baselines like Habermas Machine, and experiments validate token-level credit assignment and scalability via token chunking. Overall, the framework offers a principled, analyzable path toward fair, multi-agent consensus in open-ended AI-generated text, with practical methods and clear directions for future work.

Abstract

Current frameworks for consensus statement generation with large language models lack the inherent structure needed to provide provable fairness guarantees when aggregating diverse free-form opinions. We model the task as a multi-objective, token-level Markov Decision Process (MDP), where each objective corresponds to an agent's preference. Token-level rewards for each agent are derived from their policy (e.g., a personalized language model). This approach utilizes the finding that such policies implicitly define optimal Q-functions, providing a principled way to quantify rewards at each generation step without a value function (Rafailov et al., 2024). This MDP formulation creates a formal structure amenable to analysis using principles from social choice theory. We propose two approaches grounded in social choice theory. First, we propose a stochastic generation policy guaranteed to be in the ex-ante core, extending core stability concepts from voting theory to text generation. This policy is derived from an underlying distribution over complete statements that maximizes proportional fairness (Nash Welfare). Second, for generating a single statement, we target the maximization of egalitarian welfare using search algorithms within the MDP framework. Empirically, experiments using language models to instantiate agent policies show that search guided by the egalitarian objective generates consensus statements with improved worst-case agent alignment compared to baseline methods, including the Habermas Machine (Tessler et al., 2024).

Generating Fair Consensus Statements with Social Choice on Token-Level MDPs

TL;DR

This work addresses the challenge of provably fair consensus statement generation by modeling open-ended text formation as a token-level MDP. Each agent contributes a policy that defines token-level rewards, enabling fairness notions from social choice theory to be directly integrated into generation. The authors develop two main contributions: (i) a stochastic policy that optimizes Nash Welfare to produce an ex-ante core lottery over complete statements, and (ii) deterministic, egalitarian-welfare-based search methods (finite lookahead and beam search) for producing a single consensus statement. Empirically, the approach yields improved worst-case agent alignment (lower Egalitarian Perplexity) compared with baselines like Habermas Machine, and experiments validate token-level credit assignment and scalability via token chunking. Overall, the framework offers a principled, analyzable path toward fair, multi-agent consensus in open-ended AI-generated text, with practical methods and clear directions for future work.

Abstract

Current frameworks for consensus statement generation with large language models lack the inherent structure needed to provide provable fairness guarantees when aggregating diverse free-form opinions. We model the task as a multi-objective, token-level Markov Decision Process (MDP), where each objective corresponds to an agent's preference. Token-level rewards for each agent are derived from their policy (e.g., a personalized language model). This approach utilizes the finding that such policies implicitly define optimal Q-functions, providing a principled way to quantify rewards at each generation step without a value function (Rafailov et al., 2024). This MDP formulation creates a formal structure amenable to analysis using principles from social choice theory. We propose two approaches grounded in social choice theory. First, we propose a stochastic generation policy guaranteed to be in the ex-ante core, extending core stability concepts from voting theory to text generation. This policy is derived from an underlying distribution over complete statements that maximizes proportional fairness (Nash Welfare). Second, for generating a single statement, we target the maximization of egalitarian welfare using search algorithms within the MDP framework. Empirically, experiments using language models to instantiate agent policies show that search guided by the egalitarian objective generates consensus statements with improved worst-case agent alignment compared to baseline methods, including the Habermas Machine (Tessler et al., 2024).
Paper Structure (66 sections, 3 theorems, 26 equations, 9 figures, 13 tables, 3 algorithms)

This paper contains 66 sections, 3 theorems, 26 equations, 9 figures, 13 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative Social Choice.
  4. Randomized Social Choice.
  5. Guided Decoding.
  6. Problem Setup & Preliminaries
  7. Agent Policies.
  8. Token-Level MDP
  9. Agent Utilities.
  10. Deterministic vs. Stochastic Policies
  11. The Fairness of a Deterministic Policy.
  12. The Fairness of a Stochastic Policy.
  13. Defining a Stochastic Policy in the Core
  14. Runtime.
  15. Properties of the Stochastic Policy
  16. ...and 51 more sections

Key Result

Theorem 1

Let $p_{\pi^*}$ be the distribution over $\mathcal{C}$ generated by executing the policy $\pi^*$ (defined in Definition def:induced_policy) from the initial state $s_0$. Then $p_{\pi^*} = p^*$.

Figures (9)

  • Figure 1: Illustration of the token-level generation tree. Edges represent actions (chosen tokens).
  • Figure 2: Maximum coalition improvement vs. polarization.
  • Figure 3: Spearman's correlation between model likelihood and participants’ Likert ratings vs. the fraction of the user’s opinion provided.
  • Figure 4: Per-scenario egalitarian welfare (perplexity). Lower values indicate better minimum agent utility. Striped bars indicate that the method uses search over the token-level MDP. Numerical results are reported in Table \ref{['tab:eppl_results']} in \ref{['app:additional_results']}.
  • Figure 5: Illustration of the induced stochastic policy $\Pi^*$ at state $s$. The optimal lottery $p^*$ assigns probabilities to the leaf nodes (complete paths). The probability $P^*(\mathcal{C}_{\mathcal{K}}(s))$ is the sum of $p^*(X)$ for all leaves reachable from $s$. The probability $P^*(\mathcal{C}_{\mathcal{K}}(s, k))$ is the sum for leaves reachable via action $k$. The policy $\Pi^*(s, k)$ is the conditional probability of taking action $k$.
  • ...and 4 more figures

Theorems & Definitions (9)

  • Definition 1: Ex-Ante Core
  • Definition 2: Stochastic Policy Induced by $p^*$
  • Theorem 1: Equivalence of Policy-Induced Distribution and Target Lottery
  • Corollary 1: Core Membership of Stochastic Policy
  • proof
  • Definition 3: Token chunking
  • Theorem 2: No constant-factor approximation under chunking
  • proof
  • proof