Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ALIGN: Aligned Delegation with Performance Guarantees for Multi-Agent LLM Reasoning

Tong Zhu, Baiting Chen, Jin Zhou, Hua Zhou, Sriram Sankararaman, Xiaowu Dai

TL;DR

ALIGN reframes LLM reasoning as an aligned delegation game where a principal relies on multiple agent LLMs to generate diverse candidates and selects the best via ranking feedback. The method leverages online mirror descent with negative entropy to update agent policies, yielding sublinear regret and convergence to a Nash equilibrium under fair comparison and positive alignment assumptions. Theoretical guarantees show that, with equal access to candidates, multi-agent delegation improves principal utility over single-agent generation, and empirical results across GSM8K, MATH, and GSM-Hard demonstrate robust improvements over strong baselines. The framework is training-free and robust to principal choice, highlighting practical impact for robust, scalable, and provably better reasoning with LLM ensembles. The work also identifies conditions under which alignment is necessary and demonstrates empirical validation of these conditions.

Abstract

LLMs often underperform on complex reasoning tasks when relying on a single generation-and-selection pipeline. Inference-time ensemble methods can improve performance by sampling diverse reasoning paths or aggregating multiple candidate answers, but they typically treat candidates independently and provide no formal guarantees that ensembling improves reasoning quality. We propose a novel method, Aligned Delegation for Multi-Agent LLM Reasoning (ALIGN), which formulates LLM reasoning as an aligned delegation game. In ALIGN, a principal delegates a task to multiple agents that generate candidate solutions under designed incentives, and then selects among their outputs to produce a final answer. This formulation induces structured interaction among agents while preserving alignment between agent and principal objectives. We establish theoretical guarantees showing that, under a fair comparison with equal access to candidate solutions, ALIGN provably improves expected performance over single-agent generation. Our analysis accommodates correlated candidate answers and relaxes independence assumptions that are commonly used in prior work. Empirical results across a broad range of LLM reasoning benchmarks consistently demonstrate that ALIGN outperforms strong single-agent and ensemble baselines.

ALIGN: Aligned Delegation with Performance Guarantees for Multi-Agent LLM Reasoning

TL;DR

ALIGN reframes LLM reasoning as an aligned delegation game where a principal relies on multiple agent LLMs to generate diverse candidates and selects the best via ranking feedback. The method leverages online mirror descent with negative entropy to update agent policies, yielding sublinear regret and convergence to a Nash equilibrium under fair comparison and positive alignment assumptions. Theoretical guarantees show that, with equal access to candidates, multi-agent delegation improves principal utility over single-agent generation, and empirical results across GSM8K, MATH, and GSM-Hard demonstrate robust improvements over strong baselines. The framework is training-free and robust to principal choice, highlighting practical impact for robust, scalable, and provably better reasoning with LLM ensembles. The work also identifies conditions under which alignment is necessary and demonstrates empirical validation of these conditions.

Abstract

LLMs often underperform on complex reasoning tasks when relying on a single generation-and-selection pipeline. Inference-time ensemble methods can improve performance by sampling diverse reasoning paths or aggregating multiple candidate answers, but they typically treat candidates independently and provide no formal guarantees that ensembling improves reasoning quality. We propose a novel method, Aligned Delegation for Multi-Agent LLM Reasoning (ALIGN), which formulates LLM reasoning as an aligned delegation game. In ALIGN, a principal delegates a task to multiple agents that generate candidate solutions under designed incentives, and then selects among their outputs to produce a final answer. This formulation induces structured interaction among agents while preserving alignment between agent and principal objectives. We establish theoretical guarantees showing that, under a fair comparison with equal access to candidate solutions, ALIGN provably improves expected performance over single-agent generation. Our analysis accommodates correlated candidate answers and relaxes independence assumptions that are commonly used in prior work. Empirical results across a broad range of LLM reasoning benchmarks consistently demonstrate that ALIGN outperforms strong single-agent and ensemble baselines.
Paper Structure (21 sections, 13 theorems, 68 equations, 2 figures, 9 tables, 1 algorithm)

This paper contains 21 sections, 13 theorems, 68 equations, 2 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Motivation
  4. Aligned Delegation for Multi-Agent LLM
  5. Implementation of ALIGN
  6. Theoretical Guarantees
  7. Performance Guarantees for ALIGN
  8. Regret Analysis
  9. Convergence Guarantees
  10. Experiments
  11. Experiment Setup
  12. Experiment Results
  13. Validation for Assumption \ref{['assump:structure']}
  14. Related Works
  15. Conclusions
  16. ...and 6 more sections

Key Result

Theorem 1

Consider a single-agent problem $P$ and its multi-agent correspondence $P'$ with $N$ agents. Then under Assumption assump:structure, we have,

Figures (2)

  • Figure 1: Overview of ALIGN: multiple agents propose answers along distinct reasoning paths and receive utility from the principal’s ranking feedback. This setup incentivizes high-confidence outputs while promoting alignment with the principal.
  • Figure 2: Accuracy comparison across benchmark datasets for each method.

Theorems & Definitions (13)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Lemma 1: kleinberg2018delegated
  • Lemma 2: shin2023delegating
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Lemma 6
  • Lemma 7
  • ...and 3 more