Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Impact of Decentralized Learning on Player Utilities in Stackelberg Games

Kate Donahue, Nicole Immorlica, Meena Jagadeesan, Brendan Lucier, Aleksandrs Slivkins

TL;DR

This paper analyzes decentralized Stackelberg learning with two autonomous bandit learners and misaligned utilities. It proves that standard Stackelberg benchmarks are unachievable under learning, motivating the gamma-tolerant benchmarks that tolerate suboptimal behavior by the other player. The authors introduce algorithms (notably ExploreThenUCB) that achieve near-optimal $O(T^{2/3})$ regret for both players against these benchmarks, and they prove a matching lower bound, establishing near-optimality. In relaxed settings that assume continuity of utilities or weaken the benchmark, they show $O(\sqrt{T})$ regret is attainable, enabling faster joint learning. The work offers a principled framework and concrete algorithms for understanding and designing two-agent sequential learning systems with societal relevance such as chatbots and recommender systems.

Abstract

When deployed in the world, a learning agent such as a recommender system or a chatbot often repeatedly interacts with another learning agent (such as a user) over time. In many such two-agent systems, each agent learns separately and the rewards of the two agents are not perfectly aligned. To better understand such cases, we examine the learning dynamics of the two-agent system and the implications for each agent's objective. We model these systems as Stackelberg games with decentralized learning and show that standard regret benchmarks (such as Stackelberg equilibrium payoffs) result in worst-case linear regret for at least one player. To better capture these systems, we construct a relaxed regret benchmark that is tolerant to small learning errors by agents. We show that standard learning algorithms fail to provide sublinear regret, and we develop algorithms to achieve near-optimal $O(T^{2/3})$ regret for both players with respect to these benchmarks. We further design relaxed environments under which faster learning ($O(\sqrt{T})$) is possible. Altogether, our results take a step towards assessing how two-agent interactions in sequential and decentralized learning environments affect the utility of both agents.

Impact of Decentralized Learning on Player Utilities in Stackelberg Games

TL;DR

This paper analyzes decentralized Stackelberg learning with two autonomous bandit learners and misaligned utilities. It proves that standard Stackelberg benchmarks are unachievable under learning, motivating the gamma-tolerant benchmarks that tolerate suboptimal behavior by the other player. The authors introduce algorithms (notably ExploreThenUCB) that achieve near-optimal regret for both players against these benchmarks, and they prove a matching lower bound, establishing near-optimality. In relaxed settings that assume continuity of utilities or weaken the benchmark, they show regret is attainable, enabling faster joint learning. The work offers a principled framework and concrete algorithms for understanding and designing two-agent sequential learning systems with societal relevance such as chatbots and recommender systems.

Abstract

When deployed in the world, a learning agent such as a recommender system or a chatbot often repeatedly interacts with another learning agent (such as a user) over time. In many such two-agent systems, each agent learns separately and the rewards of the two agents are not perfectly aligned. To better understand such cases, we examine the learning dynamics of the two-agent system and the implications for each agent's objective. We model these systems as Stackelberg games with decentralized learning and show that standard regret benchmarks (such as Stackelberg equilibrium payoffs) result in worst-case linear regret for at least one player. To better capture these systems, we construct a relaxed regret benchmark that is tolerant to small learning errors by agents. We show that standard learning algorithms fail to provide sublinear regret, and we develop algorithms to achieve near-optimal regret for both players with respect to these benchmarks. We further design relaxed environments under which faster learning () is possible. Altogether, our results take a step towards assessing how two-agent interactions in sequential and decentralized learning environments affect the utility of both agents.
Paper Structure (81 sections, 50 theorems, 169 equations, 10 tables)

This paper contains 81 sections, 50 theorems, 169 equations, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model and assumptions
  4. Interaction between players
  5. Information structures.
  6. Measuring regret
  7. Real-world examples
  8. Assumptions on the follower's algorithm $\texttt{ALG}_2$
  9. Stackelberg value is unachievable
  10. $\gamma$-tolerant benchmark and regret bounds
  11. $\gamma$-tolerant benchmark
  12. Linear regret for ExploreThenCommit
  13. ExploreThenCommit($E, \mathcal{C})$ (Algorithm \ref{['algo:explorethencommit']}).
  14. Warmup Algorithm
  15. ExploreThenCommitThrowOut($E, E', \mathcal{C})$ (Algorithm \ref{['algo:explorethencommitthrowout']}).
  16. ...and 66 more sections

Key Result

Theorem 3.1

Consider $\text{StrongDSG}$s or $\text{WeakDSG}$s. For any algorithms $\texttt{ALG}_1$ and $\texttt{ALG}_2$, there exists an instance $\mathcal{I}^*$ with $|\mathcal{A}| = |\mathcal{B}| = 2$ where at least one of the players incurs linear regret with respect to the Stackleberg benchmarks $\beta_{1}^

Theorems & Definitions (95)

  • Example 2.1: User-chatbot interaction
  • Example 2.2: User-recommender system interaction
  • Theorem 3.1
  • proof : Proof sketch of Theorem \ref{['thrm:worstlinear']}
  • Definition 4.1
  • Example 4.2
  • Proposition 4.2
  • proof : Proof sketch of Proposition \ref{['prop:decentralizedlower']}
  • Theorem 4.3
  • proof : Proof sketch for Theorem \ref{['thm:ETCETC']}
  • ...and 85 more