Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning

TL;DR

HOP addresses few-shot adaptation to unseen co-players in mixed-motive multi-agent environments by coupling a hierarchical opponent modeling module (goal inference and goal-conditioned policies) with a planning module that uses Monte Carlo Tree Search guided by the inferred models. Beliefs about others' goals are updated both within and across episodes, enabling fast corrections to changing co-player behavior, while planning samples co-player goals from these beliefs to compute robust best responses. Empirical results in Markov Stag-Hunt and Markov Snowdrift demonstrate superior few-shot adaptation and strong self-play performance, along with emergent social intelligence such as self-organized cooperation. The work provides a scalable framework for cooperative-competitive settings and highlights the practical potential of integrating ToM-inspired inference with principled planning in complex, uncertain multi-agent domains.

Abstract

Despite the recent successes of multi-agent reinforcement learning (MARL) algorithms, efficiently adapting to co-players in mixed-motive environments remains a significant challenge. One feasible approach is to hierarchically model co-players' behavior based on inferring their characteristics. However, these methods often encounter difficulties in efficient reasoning and utilization of inferred information. To address these issues, we propose Hierarchical Opponent modeling and Planning (HOP), a novel multi-agent decision-making algorithm that enables few-shot adaptation to unseen policies in mixed-motive environments. HOP is hierarchically composed of two modules: an opponent modeling module that infers others' goals and learns corresponding goal-conditioned policies, and a planning module that employs Monte Carlo Tree Search (MCTS) to identify the best response. Our approach improves efficiency by updating beliefs about others' goals both across and within episodes and by using information from the opponent modeling module to guide planning. Experimental results demonstrate that in mixed-motive environments, HOP exhibits superior few-shot adaptation capabilities when interacting with various unseen agents, and excels in self-play scenarios. Furthermore, the emergence of social intelligence during our experiments underscores the potential of our approach in complex multi-agent environments.

Figures (6)

• Figure 1: Overview of HOP. HOP consists of an opponent modeling module and a planning module. The opponent modeling module models the behavior of co-players by inferring co-players' goals and learning their goal-conditioned policies. Estimated behavior is then fed to the planning module to select a rewarding action for the focal agent.
• Figure 2: Overview of Markov Stag-Hunt and Markov Snowdrift. There are four agents, represented by colored circles, in each paradigm. (a) Agents catch prey for reward. A stag with a reward of $10$ requires at least two agents to hunt together. One agent can hunt a hare with a reward of $1$. (b) Everyone gets a reward of $6$ when an agent removes a snowdrift. When a snowdrift is removed, removers share the cost of $4$ evenly.
• Figure 3: Schelling diagrams for (a) MSH-4h1s, (b) MSH-4h2s, and (c) MSG.
• Figure 4: Self-play performance of HOP and baseline algorithms. Shown is the average reward in the self-play training phase.
• Figure 5: Visualization of HOP's belief in adaptation to three defectors in MSH. Every blue-filled circle represents HOP's inferred probability (i.e., belief) that a co-player hunts stags.