Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incentivized Learning in Principal-Agent Bandit Games

Antoine Scheid, Daniil Tiapkin, Etienne Boursier, Aymeric Capitaine, El Mahdi El Mhamdi, Eric Moulines, Michael I. Jordan, Alain Durmus

Abstract

This work considers a repeated principal-agent bandit game, where the principal can only interact with her environment through the agent. The principal and the agent have misaligned objectives and the choice of action is only left to the agent. However, the principal can influence the agent's decisions by offering incentives which add up to his rewards. The principal aims to iteratively learn an incentive policy to maximize her own total utility. This framework extends usual bandit problems and is motivated by several practical applications, such as healthcare or ecological taxation, where traditionally used mechanism design theories often overlook the learning aspect of the problem. We present nearly optimal (with respect to a horizon $T$) learning algorithms for the principal's regret in both multi-armed and linear contextual settings. Finally, we support our theoretical guarantees through numerical experiments.

Incentivized Learning in Principal-Agent Bandit Games

Abstract

This work considers a repeated principal-agent bandit game, where the principal can only interact with her environment through the agent. The principal and the agent have misaligned objectives and the choice of action is only left to the agent. However, the principal can influence the agent's decisions by offering incentives which add up to his rewards. The principal aims to iteratively learn an incentive policy to maximize her own total utility. This framework extends usual bandit problems and is motivated by several practical applications, such as healthcare or ecological taxation, where traditionally used mechanism design theories often overlook the learning aspect of the problem. We present nearly optimal (with respect to a horizon ) learning algorithms for the principal's regret in both multi-armed and linear contextual settings. Finally, we support our theoretical guarantees through numerical experiments.
Paper Structure (19 sections, 23 theorems, 86 equations, 2 figures, 3 tables, 5 algorithms)

This paper contains 19 sections, 23 theorems, 86 equations, 2 figures, 3 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Armed Principal-Agent Learning
  4. Setup.
  5. Optimal incentives.
  6. Contextual Principal-Agent Learning
  7. Experiments
  8. Lower Bounds
  9. Conclusion and Possible Extensions
  10. Notation
  11. Experimental details
  12. Regret Bound for Non-Contextual Setting
  13. Regret Bound for the Contextual Setting
  14. Technical lemmas
  15. Proof of \ref{['theorem:regret_bound_contextual']}
  16. ...and 4 more sections

Key Result

Lemma 1

For any $T \in \mathbb{N}$, the regret of any algorithm on our problem instance can be written as

Figures (2)

  • Figure 1: Illustration of a case where the volume $\mathcal{S}_0$ is cut along a direction $w_1$ to give a new confidence set $\mathcal{S}_1$; while the diameter is not reduced along the directions $v_1$ nor $v_2$.
  • Figure 2: Cumulative regret for different algorithms on a $5$ arms instance.

Theorems & Definitions (40)

  • Lemma 1
  • Theorem 1
  • Corollary 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Theorem 2
  • Lemma 6
  • Corollary 2
  • ...and 30 more