Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoding Rewards in Competitive Games: Inverse Game Theory with Entropy Regularization

Junyi Liao, Zihan Zhu, Ethan Fang, Zhuoran Yang, Vahid Tarokh

TL;DR

This work tackles the inverse problem of recovering unknown reward functions in entropy-regularized two-player zero-sum games and Markov games. It establishes identifiability under a linear payoff assumption via the quantal response equilibrium (QRE) and develops a two-step estimation approach that constructs confidence sets for feasible reward parameters, with finite-sample guarantees. The framework extends to dynamic settings by incorporating transition learning and plug-in Bellman equations, yielding sample-efficient recovery of reward functions and Q-functions, with both frequency- and maximum-likelihood-based QRE estimators analyzed. Numerical experiments in static matrix games and dynamic Markov games demonstrate consistent recovery of reward structures and accurate QRE predictions, validating the method's practical relevance for competitive decision-making and policy auditing.

Abstract

Estimating the unknown reward functions driving agents' behaviors is of central interest in inverse reinforcement learning and game theory. To tackle this problem, we develop a unified framework for reward function recovery in two-player zero-sum matrix games and Markov games with entropy regularization, where we aim to reconstruct the underlying reward functions given observed players' strategies and actions. This task is challenging due to the inherent ambiguity of inverse problems, the non-uniqueness of feasible rewards, and limited observational data coverage. To address these challenges, we establish the reward function's identifiability using the quantal response equilibrium (QRE) under linear assumptions. Building upon this theoretical foundation, we propose a novel algorithm to learn reward functions from observed actions. Our algorithm works in both static and dynamic settings and is adaptable to incorporate different methods, such as Maximum Likelihood Estimation (MLE). We provide strong theoretical guarantees for the reliability and sample efficiency of our algorithm. Further, we conduct extensive numerical studies to demonstrate the practical effectiveness of the proposed framework, offering new insights into decision-making in competitive environments.

Decoding Rewards in Competitive Games: Inverse Game Theory with Entropy Regularization

TL;DR

This work tackles the inverse problem of recovering unknown reward functions in entropy-regularized two-player zero-sum games and Markov games. It establishes identifiability under a linear payoff assumption via the quantal response equilibrium (QRE) and develops a two-step estimation approach that constructs confidence sets for feasible reward parameters, with finite-sample guarantees. The framework extends to dynamic settings by incorporating transition learning and plug-in Bellman equations, yielding sample-efficient recovery of reward functions and Q-functions, with both frequency- and maximum-likelihood-based QRE estimators analyzed. Numerical experiments in static matrix games and dynamic Markov games demonstrate consistent recovery of reward structures and accurate QRE predictions, validating the method's practical relevance for competitive decision-making and policy auditing.

Abstract

Estimating the unknown reward functions driving agents' behaviors is of central interest in inverse reinforcement learning and game theory. To tackle this problem, we develop a unified framework for reward function recovery in two-player zero-sum matrix games and Markov games with entropy regularization, where we aim to reconstruct the underlying reward functions given observed players' strategies and actions. This task is challenging due to the inherent ambiguity of inverse problems, the non-uniqueness of feasible rewards, and limited observational data coverage. To address these challenges, we establish the reward function's identifiability using the quantal response equilibrium (QRE) under linear assumptions. Building upon this theoretical foundation, we propose a novel algorithm to learn reward functions from observed actions. Our algorithm works in both static and dynamic settings and is adaptable to incorporate different methods, such as Maximum Likelihood Estimation (MLE). We provide strong theoretical guarantees for the reliability and sample efficiency of our algorithm. Further, we conduct extensive numerical studies to demonstrate the practical effectiveness of the proposed framework, offering new insights into decision-making in competitive environments.
Paper Structure (75 sections, 23 theorems, 304 equations, 2 figures, 1 table, 3 algorithms)

This paper contains 75 sections, 23 theorems, 304 equations, 2 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Motivating Applications
  3. Economics and Marketing.
  4. Cybersecurity.
  5. Operations Research.
  6. Major Contributions
  7. Related Work
  8. Zero-Sum Markov Games.
  9. Inverse Optimization and Inverse Reinforcement Learning (IRL).
  10. Entropy Regularization in RL and Games.
  11. Entropy-Regularized Zero-Sum Matrix Games
  12. Problem Formulation
  13. Entropy-Regularized Two-Player Zero-Sum Matrix Game.
  14. Strong Identifiability
  15. Partial Identifiability
  16. ...and 60 more sections

Key Result

Proposition 2.2

Under Assumption a3.1, there is a unique $\theta\in\mathbb{R}^d$ such that $Q(a,b) = \langle\phi(a,b),\theta\rangle$ (i.e. $\theta = \theta^*$) for all $(a,b)\in\mathcal{A}\times\mathcal{B}$ if and only if the QRE satisfies the rank condition that

Figures (2)

  • Figure 1: The results of numerical simulation on zero-sum matrix games. Both X and Y axes are log-scaled. The X-axis represents the sample size from $10^3$ to $10^6$. The Y-axis represents (a,b) the error $\Vert\widehat{\theta}-\theta^*\Vert$ of the estimate $\widehat{\theta}$; (c,d) The Y-axis represents the error $\Vert\widehat{Q}-Q^*\Vert_{\mathrm{F}}$ of the reward function $\widehat{Q}$; (e,f) The Y-axis represents the error $\mathrm{TV}(\widehat{\mu},\mu^*)+\mathrm{TV}(\widehat{\nu},\nu^*)$. We repeat 100 experiments for each sample size and plot 95% confidence interval for the error.
  • Figure 2: The result of simulation in entropy-regularized zero-sum Markov games. (a) The reconstruction error of the reward functions $(\widehat{r}_h)_{h=1}^6$. The X-axis represents the time step $h$ from $1$ to $6$, while the Y-axis represents the error $\Vert\widehat{r}_h-r^*_h\Vert_{\mathrm{F}}$ of the reward function $\widehat{r}$. (b) The discrepancy between the QRE $(\widehat{\mu},\widehat{\nu})$ corresponding to the estimated reward functions $(\widehat{r}_h)_{h=1}^6$ and the true QRE $(\mu^*,\nu^*)$. The X-axis represents the time step $h$ from $1$ to $6$, while the Y-axis represents the errors $\mathrm{TV}(\widehat{\mu}_h,\mu^*_h)+\mathrm{TV}(\widehat{\nu}_h,\nu^*_h)$. We repeat 100 experiments for each sample size and plot 95% confidence interval for the error.

Theorems & Definitions (30)

  • Proposition 2.2: Sufficient and necessary condition for strong identifiability
  • Theorem 2.3: Parameter estimation error
  • Theorem 2.4: Finite sample error bound
  • Definition 2.5: Hausdorff distance
  • Lemma 2.6: Construction error
  • Theorem 2.7: Convergence of confidence set
  • Remark 2.8: Convergence of payoff function
  • Definition 3.1: Quantal response equilibrium
  • Definition 3.2: identified reward set
  • Proposition 3.4: Strong Q-identifiability
  • ...and 20 more