The Benefits of Power Regularization in Cooperative Reinforcement Learning
Michelle Li, Michael Dennis
TL;DR
This work introduces power regularization for cooperative MARL by defining a 1-step adversarial power measure and a linear trade-off objective $U_i(\pi|s)=U_i^{task}(\pi|s)+\\lambda U_i^{power}(\pi|s)$ to discourage power concentration. It proves that power-regularizing equilibria exist for any $\\lambda$ by mapping to a $p$-adversarial game and applying Nash's theorem, and provides two training algorithms, SBPR and PRIM, to optimize the regularized objective. Through experiments in small games and Overcooked variants, the authors show that both methods can balance task reward and power, reducing vulnerability to off-policy deviations and extreme events, with PRIM often offering superior stability at low regularization levels. The results highlight the practical value of explicitly regulating power to improve robustness and cooperative performance in multi-agent systems.
Abstract
Cooperative Multi-Agent Reinforcement Learning (MARL) algorithms, trained only to optimize task reward, can lead to a concentration of power where the failure or adversarial intent of a single agent could decimate the reward of every agent in the system. In the context of teams of people, it is often useful to explicitly consider how power is distributed to ensure no person becomes a single point of failure. Here, we argue that explicitly regularizing the concentration of power in cooperative RL systems can result in systems which are more robust to single agent failure, adversarial attacks, and incentive changes of co-players. To this end, we define a practical pairwise measure of power that captures the ability of any co-player to influence the ego agent's reward, and then propose a power-regularized objective which balances task reward and power concentration. Given this new objective, we show that there always exists an equilibrium where every agent is playing a power-regularized best-response balancing power and task reward. Moreover, we present two algorithms for training agents towards this power-regularized objective: Sample Based Power Regularization (SBPR), which injects adversarial data during training; and Power Regularization via Intrinsic Motivation (PRIM), which adds an intrinsic motivation to regulate power to the training objective. Our experiments demonstrate that both algorithms successfully balance task reward and power, leading to lower power behavior than the baseline of task-only reward and avoid catastrophic events in case an agent in the system goes off-policy.