Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Model Selection For Learning In Strategic Environments

Tinashe Handina, Eric Mazumdar

TL;DR

It is found that strategic interactions can break the conventional view of performance improving as model classes get larger or more expressive (even with infinite data), and a new paradigm for model selection in games wherein an agent seeks to choose amongst different model classes to use as their action set in a game is proposed.

Abstract

The deployment of ever-larger machine learning models reflects a growing consensus that the more expressive the model class one optimizes over$\unicode{x2013}$and the more data one has access to$\unicode{x2013}$the more one can improve performance. As models get deployed in a variety of real-world scenarios, they inevitably face strategic environments. In this work, we consider the natural question of how the interplay of models and strategic interactions affects the relationship between performance at equilibrium and the expressivity of model classes. We find that strategic interactions can break the conventional view$\unicode{x2013}$meaning that performance does not necessarily monotonically improve as model classes get larger or more expressive (even with infinite data). We show the implications of this result in several contexts including strategic regression, strategic classification, and multi-agent reinforcement learning. In particular, we show that each of these settings admits a Braess' paradox-like phenomenon in which optimizing over less expressive model classes allows one to achieve strictly better equilibrium outcomes. Motivated by these examples, we then propose a new paradigm for model selection in games wherein an agent seeks to choose amongst different model classes to use as their action set in a game.

Understanding Model Selection For Learning In Strategic Environments

TL;DR

It is found that strategic interactions can break the conventional view of performance improving as model classes get larger or more expressive (even with infinite data), and a new paradigm for model selection in games wherein an agent seeks to choose amongst different model classes to use as their action set in a game is proposed.

Abstract

The deployment of ever-larger machine learning models reflects a growing consensus that the more expressive the model class one optimizes overand the more data one has access tothe more one can improve performance. As models get deployed in a variety of real-world scenarios, they inevitably face strategic environments. In this work, we consider the natural question of how the interplay of models and strategic interactions affects the relationship between performance at equilibrium and the expressivity of model classes. We find that strategic interactions can break the conventional viewmeaning that performance does not necessarily monotonically improve as model classes get larger or more expressive (even with infinite data). We show the implications of this result in several contexts including strategic regression, strategic classification, and multi-agent reinforcement learning. In particular, we show that each of these settings admits a Braess' paradox-like phenomenon in which optimizing over less expressive model classes allows one to achieve strictly better equilibrium outcomes. Motivated by these examples, we then propose a new paradigm for model selection in games wherein an agent seeks to choose amongst different model classes to use as their action set in a game.
Paper Structure (22 sections, 14 theorems, 55 equations, 2 figures, 2 algorithms)

This paper contains 22 sections, 14 theorems, 55 equations, 2 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions:
  3. Related work
  4. Preliminaries
  5. Non-Monotonic Scaling of Performance in Nash Settings
  6. Example 1: Multi-Agent Reinforcement Learning
  7. Example 2: Participation dynamics
  8. Online Learning for Model Selection in Games
  9. Conclusion
  10. Learner-Environment Interactions
  11. Description of learner environment settings
  12. Stationary environments and Stackelberg games with the learner leading
  13. Stackelberg games where the learner follows
  14. Example 3: Strategic Linear Regression
  15. Additional proofs
  16. ...and 7 more sections

Key Result

Theorem 3.4

For a two-player monotone game $G$ on $\Theta \times \mathcal{E}$ which satisfies Assumption game_assumptions, if the unique Nash equilibrium in $\Theta\times \mathcal{E}$, $(\theta^*, e^*)$, is not Pareto optimal then there exists a restriction of the learner's model class (i.e., a set $\Theta' \su

Figures (2)

  • Figure 1: (a.) A visual description of a 2-player Markov game in which the learner can unilaterally increase their payoff by restricting the expressivity of their policy class. (b.) the payoff of the learner at Nash in a 50-state version of this Markov game as their policy class is restricted to take the form $\pi_l(s)=[p,1-p]$ in all states $s$ for $p\in [1-\bar{p}, \bar{p}]$ for different discount factors (assumed to be the same for both players). In all cases, we see the learners' payoff broadly increase at Nash as they optimize over smaller policy classes.
  • Figure 2: The loss for the learner at their best response in a regression game as the magnitude of the environment's perturbation vector varies with the payoffs achieved at equilibrium as derived in Proposition \ref{['prop:linreg']}.

Theorems & Definitions (26)

  • Definition 2.1
  • Definition 3.1
  • Definition 3.3
  • Theorem 3.4
  • Proposition 3.6
  • Proposition 4.2
  • Proposition 4.3
  • Proposition A.1
  • proof
  • Proposition A.2
  • ...and 16 more