Contextual Bilevel Reinforcement Learning for Incentive Alignment

TL;DR

CB-RL extends bilevel reinforcement learning to a setting with contextual CMDPs and multiple followers, enabling incentive design under uncertainty. The authors introduce Hyper Policy Gradient Descent (HPGD), which estimates the hypergradient from follower trajectories, making the leader's optimization scalable and agnostic to the followers' training dynamics. They prove non-asymptotic convergence to a stationary point and propose an accelerated variant (RT-Q) when the leader can control follower training. Empirical results on Four-Rooms and macro tax design illustrate improved outer-level performance and practical applicability to RLHF, contract design, and mechanism design.

Abstract

The optimal policy in various real-world strategic decision-making problems depends both on the environmental configuration and exogenous events. For these settings, we introduce Contextual Bilevel Reinforcement Learning (CB-RL), a stochastic bilevel decision-making model, where the lower level consists of solving a contextual Markov Decision Process (CMDP). CB-RL can be viewed as a Stackelberg Game where the leader and a random context beyond the leader's control together decide the setup of many MDPs that potentially multiple followers best respond to. This framework extends beyond traditional bilevel optimization and finds relevance in diverse fields such as RLHF, tax design, reward shaping, contract theory and mechanism design. We propose a stochastic Hyper Policy Gradient Descent (HPGD) algorithm to solve CB-RL, and demonstrate its convergence. Notably, HPGD uses stochastic hypergradient estimates, based on observations of the followers' trajectories. Therefore, it allows followers to use any training procedure and the leader to be agnostic of the specific algorithm, which aligns with various real-world scenarios. We further consider the setting when the leader can influence the training of followers and propose an accelerated algorithm. We empirically demonstrate the performance of our algorithm for reward shaping and tax design.

Figures (21)

• Figure 1: Four-Rooms State Space and Performance. Left: $S$ denotes the start state while $G^1$ and $G^2$ denote goal states that are considered separate tasks. $+1$ denotes the target cell to which the upper-level aims to steer the lower-level MDP. Right: HPGD escapes local optima achieving higher performance than comparison algorithms.
• Figure 2: Reward penalties given to the lower-level agent in each state of the Four-Rooms problem optimized by the HPGD, AMD, and Zero-Order, respectively. HPGD efficiently steers the lower-level MDP when the task is to reach $G^1$ while others are only successful in the case of $G^2$.
• Figure 3: Upper-level objective values, $F$, over the number of outer iterations for hyperparameters $\lambda = 0.001$ and $\beta = 3.0$
• Figure 4: Upper-level objective values, $F$, over the number of outer iterations for hyperparameters $\lambda = 0.001$ and $\beta = 5.0$
• Figure 5: Upper-level objective values, $F$, over the number of outer iterations for hyperparameters $\lambda = 0.003$ and $\beta = 1.0$

Theorems & Definitions (23)

• Theorem 1
• Theorem 2
• Theorem 3
• Proposition 1
• Theorem 4: Improved iteration complexity using RT-Q
• proof
• proof
• proof
• proof
• proof