Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Regularized Actor-Critic Algorithm for Bi-Level Reinforcement Learning

Sihan Zeng, Sujay Bhatt, Sumitra Ganesh, Alec Koppel

TL;DR

This work tackles bi-level reinforcement learning where an upper-level variable $x$ shapes the lower-level reward function, and the outer objective $f$ depends on the optimal lower-level policy. It introduces entropy regularization with weight $\tau$ to induce a gradient-dominating structure in the lower-level RL and forms a single-loop, penalty-based algorithm that estimates the upper-level hyper-gradient using only first-order information, avoiding costly second-order terms. A key contribution is the finite-time and finite-sample convergence guarantee to a stationary point of the original unregularized bi-level RL objective, achieved via a novel lower-level residual analysis under a Polyak–Łojasiewicz-type condition and a careful five-time-scale stochastic approximation argument; the regularization decays to track the unregularized problem. The paper validates the approach on GridWorld and RLHF-like tweet generation tasks, showing improved convergence and solution quality over baselines and highlighting the practical impact for reward design and policy optimization in complex RL settings.

Abstract

We study a structured bi-level optimization problem where the upper-level objective is a smooth function and the lower-level problem is policy optimization in a Markov decision process (MDP). The upper-level decision variable parameterizes the reward of the lower-level MDP, and the upper-level objective depends on the optimal induced policy. Existing methods for bi-level optimization and RL often require second-order information, impose strong regularization at the lower level, or inefficiently use samples through nested-loop procedures. In this work, we propose a single-loop, first-order actor-critic algorithm that optimizes the bi-level objective via a penalty-based reformulation. We introduce into the lower-level RL objective an attenuating entropy regularization, which enables asymptotically unbiased upper-level hyper-gradient estimation without solving the unregularized RL problem exactly. We establish the finite-time and finite-sample convergence of the proposed algorithm to a stationary point of the original, unregularized bi-level optimization problem through a novel lower-level residual analysis under a special type of Polyak-Lojasiewicz condition. We validate the performance of our method through experiments on a GridWorld goal position problem and on happy tweet generation through reinforcement learning from human feedback (RLHF).

A Regularized Actor-Critic Algorithm for Bi-Level Reinforcement Learning

TL;DR

This work tackles bi-level reinforcement learning where an upper-level variable shapes the lower-level reward function, and the outer objective depends on the optimal lower-level policy. It introduces entropy regularization with weight to induce a gradient-dominating structure in the lower-level RL and forms a single-loop, penalty-based algorithm that estimates the upper-level hyper-gradient using only first-order information, avoiding costly second-order terms. A key contribution is the finite-time and finite-sample convergence guarantee to a stationary point of the original unregularized bi-level RL objective, achieved via a novel lower-level residual analysis under a Polyak–Łojasiewicz-type condition and a careful five-time-scale stochastic approximation argument; the regularization decays to track the unregularized problem. The paper validates the approach on GridWorld and RLHF-like tweet generation tasks, showing improved convergence and solution quality over baselines and highlighting the practical impact for reward design and policy optimization in complex RL settings.

Abstract

We study a structured bi-level optimization problem where the upper-level objective is a smooth function and the lower-level problem is policy optimization in a Markov decision process (MDP). The upper-level decision variable parameterizes the reward of the lower-level MDP, and the upper-level objective depends on the optimal induced policy. Existing methods for bi-level optimization and RL often require second-order information, impose strong regularization at the lower level, or inefficiently use samples through nested-loop procedures. In this work, we propose a single-loop, first-order actor-critic algorithm that optimizes the bi-level objective via a penalty-based reformulation. We introduce into the lower-level RL objective an attenuating entropy regularization, which enables asymptotically unbiased upper-level hyper-gradient estimation without solving the unregularized RL problem exactly. We establish the finite-time and finite-sample convergence of the proposed algorithm to a stationary point of the original, unregularized bi-level optimization problem through a novel lower-level residual analysis under a special type of Polyak-Lojasiewicz condition. We validate the performance of our method through experiments on a GridWorld goal position problem and on happy tweet generation through reinforcement learning from human feedback (RLHF).
Paper Structure (40 sections, 24 theorems, 255 equations, 2 figures, 1 table, 2 algorithms)

This paper contains 40 sections, 24 theorems, 255 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. FORMULATION
  4. Entropy Regularization
  5. ALGORITHM DEVELOPMENT
  6. Preliminaries -- Penalty Reformulation
  7. Single-Loop Actor-Critic Algorithm
  8. CONVERGENCE ANALYSIS
  9. EXPERIMENTAL RESULTS
  10. CONCLUSION & FUTURE WORK
  11. Frequently Used Notations, Equations, and Inequalities
  12. Applications of Bi-Level Reinforcement Learning
  13. Reward Shaping
  14. Inverse Reinforcement Learning
  15. Proof of Theorems
  16. ...and 25 more sections

Key Result

Lemma 1

Let $\pi^\star(x)$ denote the optimal policy for the unregularized MDP with the largest (weighted) entropy Then, under Assumption assump:exploration, it holds that $\pi^\star(x)$ is unique for all $x$ and is the limit point of $\{\pi_\tau^\star(x)\}_\tau$

Figures (2)

  • Figure 1: Algorithm Performance on GridWorld Goal Placement (Left) and Language Model Finetuning (Right)
  • Figure 2: GridWorld Illustration. The red flag is the goal in the lower-level MDP set by the upper-level decision variable. A state further away from the goal incurs a negative reward with higher magnitude. The green circle indicates the center of the grid, which defines a component of the upper-level objective.

Theorems & Definitions (26)

  • Remark 1
  • Lemma 1
  • Remark 2
  • Theorem 1
  • Theorem 2
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Lemma 6
  • ...and 16 more