Preference-Based Multi-Agent Reinforcement Learning: Data Coverage and Algorithmic Techniques

TL;DR

PbMARL addresses learning a Nash equilibrium from offline preference data in general-sum Markov games, tackling sparse feedback with a focus on unilateral data coverage. The work proves that unilateral coverage is necessary and sufficient for recovering an approximate Nash equilibrium, and it introduces a practical two-phase pipeline that learns a reward model from preferences and applies offline MARL with distribution-based pessimism. Two key techniques—time-axis MSE reward regularization and dataset-density-based pessimism—improve reward learning and training stability across diverse environments, as validated by extensive ablations and dataset diversity studies. The results provide a foundational framework for preference-based multi-agent systems and point toward future work integrating LLMs to further enhance coordination and alignment in complex settings.

Abstract

We initiate the study of Preference-Based Multi-Agent Reinforcement Learning (PbMARL), exploring both theoretical foundations and empirical validations. We define the task as identifying the Nash equilibrium from a preference-only offline dataset in general-sum games, a problem marked by the challenge of sparse feedback signals. Our theory establishes the upper complexity bounds for Nash Equilibrium in effective PbMARL, demonstrating that single-policy coverage is inadequate and highlighting the importance of unilateral dataset coverage. These theoretical insights are verified through comprehensive experiments. To enhance the practical performance, we further introduce two algorithmic techniques. (1) We propose a Mean Squared Error (MSE) regularization along the time axis to achieve a more uniform reward distribution and improve reward learning outcomes. (2) We propose an additional penalty based on the distribution of the dataset to incorporate pessimism, improving stability and effectiveness during training. Our findings underscore the multifaceted approach required for PbMARL, paving the way for effective preference-based multi-agent systems.

Figures (4)

• Figure 1: The overall pipeline of offline PbMARL. $\mathcal{D}$ is the preference dataset where $\tau_i, \tau_i'$ are trajectories and $\mathbf{y}_i \in \{1,-1\}^m$ indicates which trajectory is preferred by each agent. $r_\phi$ is the learned reward. $\pi_{b}$ is the learned reference policy using imitation learning.
• Figure 2: (a) Averaged reward predictions and ground truth of a trajectory sample on spread-v3. (b) Standardized reward predictions and ground truth of a trajectory sample in reference-v3. When trained with expert data only (b1), $\phi$ experiences a mode collapse, failing to give informative signals. Reward function trained without regularization (b2) shows spiky patterns and tends to accumulate predictions at certain time steps when trained with less diversified datasets as (a). Our method with diversified dataset (b3) gives predictions that approximate the ground truth well.
• Figure 3: Reward model training curves on Spread-v3 Diversified dataset. Extra positive MSE regularization results in lower final training loss.
• Figure 4: Predicted rewards and ground truth (both standardized) in all environments. Our method with diversified dataset and reward regularization gives predictions that approximate the ground truth the best.

Theorems & Definitions (17)

• Definition 1
• Theorem 1
• Theorem 2
• Theorem 3
• proof
• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3