Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF

TL;DR

This document provides a comprehensive guide to the SIAM LaTeX style, covering class options, front matter, cross-referencing, and typesetting facilities for mathematics, theorems, tables, figures, and algorithms. It explains how to manage supplements, PDF bookmarks, and bibliographic integration, enabling SIAM-compliant manuscript preparation. The content serves as a practical template and reference for authors preparing submissions consistent with SIAM conventions. Overall, it ensures robust formatting, navigation, and presentation across the main document and any supplementary materials.

Abstract

Bilevel optimization has been recently applied to many machine learning tasks. However, their applications have been restricted to the supervised learning setting, where static objective functions with benign structures are considered. But bilevel problems such as incentive design, inverse reinforcement learning (RL), and RL from human feedback (RLHF) are often modeled as dynamic objective functions that go beyond the simple static objective structures, which pose significant challenges of using existing bilevel solutions. To tackle this new class of bilevel problems, we introduce the first principled algorithmic framework for solving bilevel RL problems through the lens of penalty formulation. We provide theoretical studies of the problem landscape and its penalty-based (policy) gradient algorithms. We demonstrate the effectiveness of our algorithms via simulations in the Stackelberg Markov game, RL from human feedback and incentive design.

Figures (2)

• Figure 1: Example figure using external image files.
• Figure 2: Example PGFPLOTS figure.

Theorems & Definitions (4)

• theorem 1: Mean Value Theorem
• corollary 1
• proof
• proof : Proof of main theorem