Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ELA: Exploited Level Augmentation for Offline Learning in Zero-Sum Games

Shiqi Lei, Kanghoon Lee, Linjing Li, Jinkyoo Park, Jiachen Li

TL;DR

This study introduces a novel approach using unsupervised learning techniques to estimate the exploited level (EL) of each trajectory from the offline dataset of zero-sum games made by diverse demonstrators, and integrates it into offline learning to maximize the influence of the dominant strategy.

Abstract

Offline learning has become widely used due to its ability to derive effective policies from offline datasets gathered by expert demonstrators without interacting with the environment directly. Recent research has explored various ways to enhance offline learning efficiency by considering the characteristics (e.g., expertise level or multiple demonstrators) of the dataset. However, a different approach is necessary in the context of zero-sum games, where outcomes vary significantly based on the strategy of the opponent. In this study, we introduce a novel approach that uses unsupervised learning techniques to estimate the exploited level of each trajectory from the offline dataset of zero-sum games made by diverse demonstrators. Subsequently, we incorporate the estimated exploited level into the offline learning to maximize the influence of the dominant strategy. Our method enables interpretable exploited level estimation in multiple zero-sum games and effectively identifies dominant strategy data. Also, our exploited level augmented offline learning significantly enhances the original offline learning algorithms including imitation learning and offline reinforcement learning for zero-sum games.

ELA: Exploited Level Augmentation for Offline Learning in Zero-Sum Games

TL;DR

This study introduces a novel approach using unsupervised learning techniques to estimate the exploited level (EL) of each trajectory from the offline dataset of zero-sum games made by diverse demonstrators, and integrates it into offline learning to maximize the influence of the dominant strategy.

Abstract

Offline learning has become widely used due to its ability to derive effective policies from offline datasets gathered by expert demonstrators without interacting with the environment directly. Recent research has explored various ways to enhance offline learning efficiency by considering the characteristics (e.g., expertise level or multiple demonstrators) of the dataset. However, a different approach is necessary in the context of zero-sum games, where outcomes vary significantly based on the strategy of the opponent. In this study, we introduce a novel approach that uses unsupervised learning techniques to estimate the exploited level of each trajectory from the offline dataset of zero-sum games made by diverse demonstrators. Subsequently, we incorporate the estimated exploited level into the offline learning to maximize the influence of the dominant strategy. Our method enables interpretable exploited level estimation in multiple zero-sum games and effectively identifies dominant strategy data. Also, our exploited level augmented offline learning significantly enhances the original offline learning algorithms including imitation learning and offline reinforcement learning for zero-sum games.
Paper Structure (20 sections, 2 theorems, 30 equations, 8 figures)

This paper contains 20 sections, 2 theorems, 30 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. An Intuition of Exploited Level
  5. Problem Formulation
  6. Method
  7. Learning Strategy Representation
  8. Exploited Level Estimator
  9. EL Augmentation for Offline Learning
  10. Experiments
  11. Experiment Settings
  12. Representation and EL Estimation
  13. EL Augmented Offline Learning
  14. Conclusions
  15. The Proof of Proposition \ref{['prop1']}
  16. ...and 5 more sections

Key Result

Proposition 6.1

If $\tau(\pi)$ is a distribution over $\Pi$, and $E$ is defined as exploitability, then we have

Figures (8)

  • Figure 1: Illustration of ELA (Exploited Level Augmentation) for offline learning in zero-sum games. It learns the exploited level in an unsupervised manner and prioritizes trajectories created by the dominant behavior in offline learning.
  • Figure 2: Illustration of EL and exploitability of a strategy in a two-player zero-sum game with three pure strategies.
  • Figure 3: The basic structures of games with representation-dependent policy. (a) Games with simultaneous actions. (b) Games with sequential actions.
  • Figure 4: The network structure of the P-VRNN model.
  • Figure 5: The overall diagram of Exploited Level Augmentation (ELA) for offline learning.
  • ...and 3 more figures

Theorems & Definitions (4)

  • Proposition 6.1
  • Proposition 6.2
  • proof
  • proof