A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning
Anjie Liu, Jianhong Wang, Samuel Kaski, Jun Wang, Mengyue Yang
TL;DR
The paper addresses the difficulty of steering cooperative MARL under partial information and scalability when global human guidance is impractical. It adopts Multi-Agent Influence Diagrams ($MAID$) to model interaction paradigms and introduces a targeted intervention paradigm that acts on a single agent, realized through Pre-Strategy Intervention ($PSI$), a causal-inference based pre-policy module designed to maximize a composite outcome that combines the primary coordination goal with an additional objective. It contributes a formal framework for MARL interaction paradigms, a solvability analysis via relevance graphs and $s$-reachability, and an implementation of PSI validated in environments such as Multi-Agent Particle Environment ($MPE$) and Hanabi, with open-source code. The approach offers a scalable, principled mechanism to guide MARL through targeted influence rather than full-system guidance, enabling safer and more efficient coordination in complex, real-world settings.
Abstract
Steering cooperative multi-agent reinforcement learning (MARL) towards desired outcomes is challenging, particularly when the global guidance from a human on the whole multi-agent system is impractical in a large-scale MARL. On the other hand, designing external mechanisms (e.g., intrinsic rewards and human feedback) to coordinate agents mostly relies on empirical studies, lacking a easy-to-use research tool. In this work, we employ multi-agent influence diagrams (MAIDs) as a graphical framework to address the above issues. First, we introduce the concept of MARL interaction paradigms (orthogonal to MARL learning paradigms), using MAIDs to analyze and visualize both unguided self-organization and global guidance mechanisms in MARL. Then, we design a new MARL interaction paradigm, referred to as the targeted intervention paradigm that is applied to only a single targeted agent, so the problem of global guidance can be mitigated. In implementation, we introduce a causal inference technique, referred to as Pre-Strategy Intervention (PSI), to realize the targeted intervention paradigm. Since MAIDs can be regarded as a special class of causal diagrams, a composite desired outcome that integrates the primary task goal and an additional desired outcome can be achieved by maximizing the corresponding causal effect through the PSI. Moreover, the bundled relevance graph analysis of MAIDs provides a tool to identify whether an MARL learning paradigm is workable under the design of an MARL interaction paradigm. In experiments, we demonstrate the effectiveness of our proposed targeted intervention, and verify the result of relevance graph analysis.