Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysing the Sample Complexity of Opponent Shaping

Kitty Fung, Qizhen Zhang, Chris Lu, Jia Wan, Timon Willi, Jakob Foerster

TL;DR

This work addresses the theoretical analysis gap for Model-Free Opponent Shaping by introducing R-FOS, a tabular approximation that discretises the continuous meta-MDP and uses the $R_{MAX}$ algorithm to obtain PAC-style guarantees. The authors prove exponential sample-complexity bounds in terms of the inner state-action space size and the number of agents, showing that the learned policy closely approximates the optimal meta-policy up to a constant. They provide two formulations (Case I and Case II) corresponding to different meta-state representations and derive bounds for each, then empirically validate the scaling in the Matching Pennies environment. The results demonstrate a clear link between discretisation granularity, inner-game complexity, and sample-efficiency, offering a principled way to analyze opponent shaping in multi-agent RL and highlighting practical considerations for scaling OS methods to higher-dimensional settings.

Abstract

Learning in general-sum games often yields collectively sub-optimal results. Addressing this, opponent shaping (OS) methods actively guide the learning processes of other agents, empirically leading to improved individual and group performances in many settings. Early OS methods use higher-order derivatives to shape the learning of co-players, making them unsuitable for shaping multiple learning steps. Follow-up work, Model-free Opponent Shaping (M-FOS), addresses these by reframing the OS problem as a meta-game. In contrast to early OS methods, there is little theoretical understanding of the M-FOS framework. Providing theoretical guarantees for M-FOS is hard because A) there is little literature on theoretical sample complexity bounds for meta-reinforcement learning B) M-FOS operates in continuous state and action spaces, so theoretical analysis is challenging. In this work, we present R-FOS, a tabular version of M-FOS that is more suitable for theoretical analysis. R-FOS discretises the continuous meta-game MDP into a tabular MDP. Within this discretised MDP, we adapt the $R_{max}$ algorithm, most prominently used to derive PAC-bounds for MDPs, as the meta-learner in the R-FOS algorithm. We derive a sample complexity bound that is exponential in the cardinality of the inner state and action space and the number of agents. Our bound guarantees that, with high probability, the final policy learned by an R-FOS agent is close to the optimal policy, apart from a constant factor. Finally, we investigate how R-FOS's sample complexity scales in the size of state-action space. Our theoretical results on scaling are supported empirically in the Matching Pennies environment.

Analysing the Sample Complexity of Opponent Shaping

TL;DR

This work addresses the theoretical analysis gap for Model-Free Opponent Shaping by introducing R-FOS, a tabular approximation that discretises the continuous meta-MDP and uses the algorithm to obtain PAC-style guarantees. The authors prove exponential sample-complexity bounds in terms of the inner state-action space size and the number of agents, showing that the learned policy closely approximates the optimal meta-policy up to a constant. They provide two formulations (Case I and Case II) corresponding to different meta-state representations and derive bounds for each, then empirically validate the scaling in the Matching Pennies environment. The results demonstrate a clear link between discretisation granularity, inner-game complexity, and sample-efficiency, offering a principled way to analyze opponent shaping in multi-agent RL and highlighting practical considerations for scaling OS methods to higher-dimensional settings.

Abstract

Learning in general-sum games often yields collectively sub-optimal results. Addressing this, opponent shaping (OS) methods actively guide the learning processes of other agents, empirically leading to improved individual and group performances in many settings. Early OS methods use higher-order derivatives to shape the learning of co-players, making them unsuitable for shaping multiple learning steps. Follow-up work, Model-free Opponent Shaping (M-FOS), addresses these by reframing the OS problem as a meta-game. In contrast to early OS methods, there is little theoretical understanding of the M-FOS framework. Providing theoretical guarantees for M-FOS is hard because A) there is little literature on theoretical sample complexity bounds for meta-reinforcement learning B) M-FOS operates in continuous state and action spaces, so theoretical analysis is challenging. In this work, we present R-FOS, a tabular version of M-FOS that is more suitable for theoretical analysis. R-FOS discretises the continuous meta-game MDP into a tabular MDP. Within this discretised MDP, we adapt the algorithm, most prominently used to derive PAC-bounds for MDPs, as the meta-learner in the R-FOS algorithm. We derive a sample complexity bound that is exponential in the cardinality of the inner state and action space and the number of agents. Our bound guarantees that, with high probability, the final policy learned by an R-FOS agent is close to the optimal policy, apart from a constant factor. Finally, we investigate how R-FOS's sample complexity scales in the size of state-action space. Our theoretical results on scaling are supported empirically in the Matching Pennies environment.
Paper Structure (50 sections, 18 theorems, 49 equations, 2 figures, 4 tables, 1 algorithm)

This paper contains 50 sections, 18 theorems, 49 equations, 2 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Stochastic Game
  5. Markov Decision Process
  6. Model-Free Opponent Shaping
  7. The $R_\text{MAX}$ Algorithm
  8. $\varepsilon$-Nets
  9. Sample Complexity Analysis with $R_\text{MAX}$ as Meta-Agent
  10. Assumptions
  11. Step 1: Discretising the Meta-MDP
  12. Discretising the State and Action Space: Case I
  13. Discretising the State and Action Space: Case II
  14. Discretising the Transition and Reward Function
  15. Step 2: The $m$-known Discretised MDP
  16. ...and 35 more sections

Key Result

Theorem 4.11

($R_\text{MAX}$ MDP Bound pacmdp) Suppose that $0 \leq \varepsilon<\frac{1}{1-\gamma}$ and $0 \leq \delta<1$ are two real numbers and $M=\langle S, A, T, R, \gamma\rangle$ is any MDP. There exists inputs $m=m\left(\frac{1}{\varepsilon}, \frac{1}{\delta}\right)$ and $\varepsilon_1$, satisfying $m\lef timesteps (final sample complexity bound).

Figures (2)

  • Figure 1: Empirical sample complexity while varying the trajectory window $h$. We plot the meta-reward per meta-episode. To better visualise the connection with the theory results, we plot the x-axis in $\log_{16}$ scale. The reported results are the mean over 3 seeds with standard error.
  • Figure 2: $\varepsilon$-Net for $\Theta = \{\theta \in \mathbb{R}^2: \norm{\theta} \leq R\}$net

Theorems & Definitions (24)

  • Definition 3.1
  • Definition 4.9: m-Known MDP
  • Definition 4.10: Empirical m-Known discretised MDP
  • Theorem 4.11
  • Theorem 4.12
  • Theorem 4.13
  • Theorem 4.14
  • Lemma 4.15: Simulation Lemma for Continuous MDPs
  • Lemma 4.16
  • Lemma 4.17
  • ...and 14 more