Learning Local Stackelberg Equilibria from Repeated Interactions with a Learning Agent

TL;DR

This work studies repeated games between an principal and an agent employing a mean-based learning algorithm and introduces an algorithm that constitutes a Polynomial Time Approximation Scheme (PTAS) for finding an epsilon-approximate local Stackelberg equilibrium.

Abstract

Motivated by the question of how a principal can maximize its utility in repeated interactions with a learning agent, we study repeated games between an principal and an agent employing a mean-based learning algorithm. Prior work has shown that computing or even approximating the global Stackelberg value in similar settings can require an exponential number of rounds in the size of the agent's action space, making it computationally intractable. In contrast, we shift focus to the computation of local Stackelberg equilibria and introduce an algorithm that, within the smoothed analysis framework, constitutes a Polynomial Time Approximation Scheme (PTAS) for finding an epsilon-approximate local Stackelberg equilibrium. Notably, the algorithm's runtime is polynomial in the size of the agent's action space yet exponential in (1/epsilon) - a dependency we prove to be unavoidable.

Figures (3)

• Figure 1: Algorithmic components for computing local Stackelberg equilibria.
• Figure 2: LSE vs Smoothed LSE : A game where $\overline{x}$ is a smoothed LSE for small enough $\eta$ and $x^*$ is the sole LSE. $\overline{x}$ is the optimal strategy within polytopes $P_2$ and $P_3$. And $x^*$ is the optimal strategy within $P_1$. The strategy $\overline{x}$ with action of $P_1$ has utility 1 more than the strategy of $\overline{x}$ with actions of $P_2$ or $P_3$. Since $P_1$ is a very thin polytope, $\overline{x}$ is nevertheless a smoothed LSE. It is however not an LSE.
• Figure 3: An illustration of how to move the average strategies in $\mathsf{MoveOneStep}$. If $\|\overline{\boldsymbol{x}}^{(t-1)} - \mathbf{u}^{(t)}\|_1=d$, then by choosing appropriate $\boldsymbol{x}^{(t)}$ along the line segment between $\overline{\boldsymbol{x}}^{(t-1)}$ and $\mathbf{u}^{(t)}$, the principal's average strategy can move $\ell_1$ distance of $\|\overline{\boldsymbol{x}}^{(t-1)} - \overline{\boldsymbol{x}}^{(t)}\|_1\in[0,\frac{d}{t}]$ (the shaded region is achievable).

Theorems & Definitions (57)

• Definition 2.1: $(\varepsilon,\delta)$-Approximate Local Stackelberg Strategy
• Definition 2.2: Mean-based Algorithm braverman2018selling
• Theorem 3.1: Main theorem
• proof : Proof sketch of \ref{['thm:main']}.
• Proposition 3.2: Correctness of $\mathsf{OptimizeWithinPolytope}$
• proof : Proof sketch
• Theorem 3.3: Correctness of $\searchpolytope$
• Remark 3.4
• Theorem 4.1: Lower Bound
• Theorem 5.1: Lower bound on the minimum singular value