Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs

Panatchakorn Anantaprayoon, Nataliia Babina, Nima Asgharbeygi, Jad Tarifi

TL;DR

This work proposes a multi-agent negotiation-based alignment framework that aligns LLMs to Collective Agency (CA)-an existing alignment objective introduced to promote the continual expansion of agency-while simultaneously improving conflict-resolution capability.

Abstract

The alignment of large language models (LLMs) has progressed substantially in single-agent settings through paradigms such as RLHF and Constitutional AI, with recent work exploring scalable alternatives such as RLAIF and evolving alignment objectives. However, these approaches remain limited in multi-stakeholder settings, where conflicting values arise and deliberative negotiation capabilities are required. This work proposes a multi-agent negotiation-based alignment framework that aligns LLMs to Collective Agency (CA)-an existing alignment objective introduced to promote the continual expansion of agency-while simultaneously improving conflict-resolution capability. To enable scalable training, two self-play instances of the same LLM, assigned opposing personas, engage in structured turn-based dialogue to synthesize mutually beneficial solutions. We generate synthetic moral-dilemma prompts and conflicting persona pairs, and optimize the policy via RLAIF using GRPO with an external LLM reward model. While rewards are computed from CA scores assigned to the final completion, gradients are applied to dialogue tokens to directly improve deliberative interaction dynamics. Experiments show that the resulting model achieves CA alignment comparable to a single-agent baseline while substantially improving conflict-resolution performance without degrading general language capabilities. These results suggest that negotiation-driven deliberation training provides a practical path toward LLMs that better support collective decision-making in value-conflict scenarios.

Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs

TL;DR

This work proposes a multi-agent negotiation-based alignment framework that aligns LLMs to Collective Agency (CA)-an existing alignment objective introduced to promote the continual expansion of agency-while simultaneously improving conflict-resolution capability.

Abstract

The alignment of large language models (LLMs) has progressed substantially in single-agent settings through paradigms such as RLHF and Constitutional AI, with recent work exploring scalable alternatives such as RLAIF and evolving alignment objectives. However, these approaches remain limited in multi-stakeholder settings, where conflicting values arise and deliberative negotiation capabilities are required. This work proposes a multi-agent negotiation-based alignment framework that aligns LLMs to Collective Agency (CA)-an existing alignment objective introduced to promote the continual expansion of agency-while simultaneously improving conflict-resolution capability. To enable scalable training, two self-play instances of the same LLM, assigned opposing personas, engage in structured turn-based dialogue to synthesize mutually beneficial solutions. We generate synthetic moral-dilemma prompts and conflicting persona pairs, and optimize the policy via RLAIF using GRPO with an external LLM reward model. While rewards are computed from CA scores assigned to the final completion, gradients are applied to dialogue tokens to directly improve deliberative interaction dynamics. Experiments show that the resulting model achieves CA alignment comparable to a single-agent baseline while substantially improving conflict-resolution performance without degrading general language capabilities. These results suggest that negotiation-driven deliberation training provides a practical path toward LLMs that better support collective decision-making in value-conflict scenarios.
Paper Structure (57 sections, 7 equations, 5 figures, 10 tables, 1 algorithm)

This paper contains 57 sections, 7 equations, 5 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Preliminaries
  3. Problem Setting and Notation
  4. Negotiation Phase.
  5. Final Completion Generation Phase.
  6. Collective Agency: A Dynamic Alignment Objective
  7. Methodology
  8. Dataset Generation
  9. Curriculum of Moral and Practical Dilemmas.
  10. Adversarial Personas Library.
  11. Multi-Agent Alignment Loop
  12. Scalable Settings for Negotiation and Final Completion
  13. Reward Computation and Policy Update via GRPO
  14. Experiments
  15. Experimental Setup
  16. ...and 42 more sections

Figures (5)

  • Figure 1: Overview of the multi-agent negotiation-based alignment framework.
  • Figure 2: Evaluation-set training dynamics over 1,900 gradient steps (50-step running averages). (a) CA scores showing group-wise min (dashed), mean (solid), and max (dash-dotted). (b) Negotiation agreement rate. (c) Average rounds to agreement. Training-set curves are provided in Figure \ref{['fig:training-dynamics-train']}.
  • Figure 3: Cross-judge score correspondence on 100 balanced evaluation dialogues. (a) Jittered scatter plot with mean GPT-5.2 score per GPT-4o-mini level (orange line); the monotonically increasing trend confirms ordinal consistency despite differing absolute calibration. (b) Confusion heatmap with cell counts; GPT-5.2 uses a wider score range (1--4) than GPT-4o-mini (2--5), but higher training-judge scores consistently correspond to higher evaluation-judge scores.
  • Figure 4: Single-agent aligned model: training dynamics over 2,150 gradient steps. CA scores showing group-wise min (dashed), mean (solid), and max (dash-dotted). All three metrics increase steadily, with the largest gain in the max CA ($+1.1$).
  • Figure 5: Training-set dynamics over 1,900 gradient steps (50-step running averages). (a) CA scores showing group-wise min (dashed), mean (solid), and max (dash-dotted). (b) Negotiation agreement rate. (c) Average rounds to agreement.