Learning Reward Functions for Cooperative Resilience in Multi-Agent Systems

TL;DR

This work addresses how reward design influences cooperative resilience in mixed-motive multi-agent systems by learning resilience-aligned rewards from ranked trajectories. It introduces a two-step framework that ranks trajectories using a cooperative resilience metric ρ(τ) and learns a reward function Ř(s; θ) via preference-based IRL (margin-based or probabilistic) to promote resilience. Experiments in a Commons Harvest-inspired environment show that a hybrid resilience-informed reward improves cooperative resilience, resource sustainability, and reduces last-resource depletion compared with baselines like PPO and QMIX. The results demonstrate the value of reward design as a principled tool for robust, scalable cooperation in uncertain environments and point to future extensions to larger, partially observable or adversarial settings.

Abstract

Multi-agent systems often operate in dynamic and uncertain environments, where agents must not only pursue individual goals but also safeguard collective functionality. This challenge is especially acute in mixed-motive multi-agent systems. This work focuses on cooperative resilience, the ability of agents to anticipate, resist, recover, and transform in the face of disruptions, a critical yet underexplored property in Multi-Agent Reinforcement Learning. We study how reward function design influences resilience in mixed-motive settings and introduce a novel framework that learns reward functions from ranked trajectories, guided by a cooperative resilience metric. Agents are trained in a suite of social dilemma environments using three reward strategies: i) traditional individual reward; ii) resilience-inferred reward; and iii) hybrid that balance both. We explore three reward parameterizations-linear models, hand-crafted features, and neural networks, and employ two preference-based learning algorithms to infer rewards from behavioral rankings. Our results demonstrate that hybrid strategy significantly improve robustness under disruptions without degrading task performance and reduce catastrophic outcomes like resource overuse. These findings underscore the importance of reward design in fostering resilient cooperation, and represent a step toward developing robust multi-agent systems capable of sustaining cooperation in uncertain environments.

Figures (4)

• Figure 1: (a) Mixed-motive environment used throughout this study. Two agents interact in an $8 \times 8$ grid with a central apple tree containing 16 apples. (b) Overview of our proposed reward learning pipeline. This figure illustrates the full loop from data collection to policy learning.
• Figure 2: Percentage of episodes (out of 500) in which agents consumed the last remaining apple for the best configurations under each reward parameterization and optimization method. (a) Agents trained exclusively with resilience-aligned rewards. (b) Agents trained with hybrid strategy.
• Figure 3: Performance metrics over 500 episodes. (a) Cooperative resilience. (b) Average total apple consumption per episode across both agents. (c) Episode length. (d) Last-apple consumption frequency, indicating the occurrence of social dilemma failures.
• Figure 4: Position frequency maps for Agent 1 (green) and Agent 2 (purple) under four training configurations: (i) random policy, (ii) PPO with standard rewards, (iii) QMIX, and (iv) hybrid strategy. Each heatmap depicts the spatial visitation density over 500 evaluation episodes, with apple locations marked in red. Agents were randomly initialized at the start of each episode and evaluated under the same protocol with three disruption events.