Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Stackelberg Optimization via Nonlinear Control

William Brown, Christos Papadimitriou, Tim Roughgarden

TL;DR

A unified algorithmic framework for tractable regret minimization in repeated interaction problems with adaptive agents, including performative prediction, recommendations for adaptive agents, adaptive pricing of real-valued goods, and repeated gameplay against no-regret learners is introduced.

Abstract

In repeated interaction problems with adaptive agents, our objective often requires anticipating and optimizing over the space of possible agent responses. We show that many problems of this form can be cast as instances of online (nonlinear) control which satisfy \textit{local controllability}, with convex losses over a bounded state space which encodes agent behavior, and we introduce a unified algorithmic framework for tractable regret minimization in such cases. When the instance dynamics are known but otherwise arbitrary, we obtain oracle-efficient $O(\sqrt{T})$ regret by reduction to online convex optimization, which can be made computationally efficient if dynamics are locally \textit{action-linear}. In the presence of adversarial disturbances to the state, we give tight bounds in terms of either the cumulative or per-round disturbance magnitude (for \textit{strongly} or \textit{weakly} locally controllable dynamics, respectively). Additionally, we give sublinear regret results for the cases of unknown locally action-linear dynamics as well as for the bandit feedback setting. Finally, we demonstrate applications of our framework to well-studied problems including performative prediction, recommendations for adaptive agents, adaptive pricing of real-valued goods, and repeated gameplay against no-regret learners, directly yielding extensions beyond prior results in each case.

Online Stackelberg Optimization via Nonlinear Control

TL;DR

A unified algorithmic framework for tractable regret minimization in repeated interaction problems with adaptive agents, including performative prediction, recommendations for adaptive agents, adaptive pricing of real-valued goods, and repeated gameplay against no-regret learners is introduced.

Abstract

In repeated interaction problems with adaptive agents, our objective often requires anticipating and optimizing over the space of possible agent responses. We show that many problems of this form can be cast as instances of online (nonlinear) control which satisfy \textit{local controllability}, with convex losses over a bounded state space which encodes agent behavior, and we introduce a unified algorithmic framework for tractable regret minimization in such cases. When the instance dynamics are known but otherwise arbitrary, we obtain oracle-efficient regret by reduction to online convex optimization, which can be made computationally efficient if dynamics are locally \textit{action-linear}. In the presence of adversarial disturbances to the state, we give tight bounds in terms of either the cumulative or per-round disturbance magnitude (for \textit{strongly} or \textit{weakly} locally controllable dynamics, respectively). Additionally, we give sublinear regret results for the cases of unknown locally action-linear dynamics as well as for the bandit feedback setting. Finally, we demonstrate applications of our framework to well-studied problems including performative prediction, recommendations for adaptive agents, adaptive pricing of real-valued goods, and repeated gameplay against no-regret learners, directly yielding extensions beyond prior results in each case.

Paper Structure

This paper contains 44 sections, 44 theorems, 60 equations, 8 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Online control.
  4. Strategizing against learners.
  5. Nested convex optimization.
  6. Model and Preliminaries
  7. Locally Controllable Dynamics
  8. States vs. Policies
  9. No-Regret Algorithms for Locally Controllable Dynamics
  10. Nonlinear Control via Online Convex Optimization
  11. Efficient Updates for Action-Linear Dynamics
  12. Adversarial Disturbances
  13. Unknown Dynamics
  14. Bandit Feedback
  15. Applications for Online Stackelberg Optimization
  16. ...and 29 more sections

Key Result

Proposition 1

Suppose there is some $y \in \mathop{\mathrm{\mathcal{Y}}}\nolimits$ and values $\alpha, \beta > 0$ such that for all $\hat{y} \in \mathop{\mathrm{\mathcal{B}}}\nolimits_{\alpha}(y)$ and $x \in \mathop{\mathrm{\mathcal{X}}}\nolimits$, $D(x, \hat{y}) \notin \mathop{\mathrm{\mathcal{B}}}\nolimits_{\be

Theorems & Definitions (56)

  • Definition 1: Weak Local Controllability
  • Definition 2: Strong Local Controllability
  • Proposition 1
  • Proposition 2
  • Theorem 1
  • Proposition 3
  • Definition 3: Locally Action-Linear Dynamics
  • Theorem 2: Bounded Disturbances for Weak Local Controllability
  • Theorem 3: Unbounded Disturbances for Strong Local Controllability
  • Theorem 4
  • ...and 46 more