Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Constrained Reinforcement Learning under Partial Data Coverage

Kihyuk Hong, Ambuj Tewari

TL;DR

The paper studies offline constrained reinforcement learning with partial data coverage and general function approximation, aiming to maximize a primary reward while meeting thresholds on auxiliary rewards using a fixed dataset. It introduces an LP-based, oracle-efficient primal-dual algorithm (PDORL) that leverages Lagrangian decomposition and a reparameterization trick to avoid policy extraction and knowledge of the data-generating distribution, under mild realizability assumptions. The authors show that saddle points of the restricted Lagrangian yield optimal policies and derive finite-sample guarantees with an $O(\epsilon^{-2})$ dependence, matching standard offline RL rates without requiring full data coverage. The framework naturally extends to constrained MDPs (CMDPs) with multiple rewards, Slater’s condition ensuring bounded duals, and corresponding near-feasibility and near-optimality guarantees, making the approach practical for safety-critical offline settings.

Abstract

We study offline constrained reinforcement learning (RL) with general function approximation. We aim to learn a policy from a pre-collected dataset that maximizes the expected discounted cumulative reward for a primary reward signal while ensuring that expected discounted returns for multiple auxiliary reward signals are above predefined thresholds. Existing algorithms either require fully exploratory data, are computationally inefficient, or depend on an additional auxiliary function classes to obtain an $ε$-optimal policy with sample complexity $O(ε^{-2})$. In this paper, we propose an oracle-efficient primal-dual algorithm based on a linear programming (LP) formulation, achieving $O(ε^{-2})$ sample complexity under partial data coverage. By introducing a realizability assumption, our approach ensures that all saddle points of the Lagrangian are optimal, removing the need for regularization that complicated prior analyses. Through Lagrangian decomposition, our method extracts policies without requiring knowledge of the data-generating distribution, enhancing practical applicability.

Offline Constrained Reinforcement Learning under Partial Data Coverage

TL;DR

The paper studies offline constrained reinforcement learning with partial data coverage and general function approximation, aiming to maximize a primary reward while meeting thresholds on auxiliary rewards using a fixed dataset. It introduces an LP-based, oracle-efficient primal-dual algorithm (PDORL) that leverages Lagrangian decomposition and a reparameterization trick to avoid policy extraction and knowledge of the data-generating distribution, under mild realizability assumptions. The authors show that saddle points of the restricted Lagrangian yield optimal policies and derive finite-sample guarantees with an dependence, matching standard offline RL rates without requiring full data coverage. The framework naturally extends to constrained MDPs (CMDPs) with multiple rewards, Slater’s condition ensuring bounded duals, and corresponding near-feasibility and near-optimality guarantees, making the approach practical for safety-critical offline settings.

Abstract

We study offline constrained reinforcement learning (RL) with general function approximation. We aim to learn a policy from a pre-collected dataset that maximizes the expected discounted cumulative reward for a primary reward signal while ensuring that expected discounted returns for multiple auxiliary reward signals are above predefined thresholds. Existing algorithms either require fully exploratory data, are computationally inefficient, or depend on an additional auxiliary function classes to obtain an -optimal policy with sample complexity . In this paper, we propose an oracle-efficient primal-dual algorithm based on a linear programming (LP) formulation, achieving sample complexity under partial data coverage. By introducing a realizability assumption, our approach ensures that all saddle points of the Lagrangian are optimal, removing the need for regularization that complicated prior analyses. Through Lagrangian decomposition, our method extracts policies without requiring knowledge of the data-generating distribution, enhancing practical applicability.

Paper Structure

This paper contains 40 sections, 22 theorems, 123 equations, 1 figure, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Offline RL via Linear Programming with Regularization
  4. Offline Constrained RL with General Function Approximation
  5. Preliminaries
  6. Notations
  7. Unconstrained Reinforcement Learning
  8. Markov Decision Processes
  9. Policies and Value Functions
  10. Linear Programming Formulation of Unconstrained Reinforcement Learning
  11. Policy Extraction
  12. Offline Dataset
  13. Main Results for Offline Unconstrained RL
  14. Lagrangian and Saddle Point Problem
  15. Lagrangian Decomposition
  16. ...and 25 more sections

Key Result

Proposition 1

There exists a MDP $(\mathcal{S}, \mathcal{A}, r, \gamma, d_0)$ and function classes $\mathcal{U}\subseteq \mathbb{R}_+^{\mathcal{S} \times \mathcal{A}}$ and $\mathcal{V} \subseteq \mathbb{R}^{\mathcal{S}}$ with $\mu^\ast \in \mathcal{U}$ and $V^\ast \in \mathcal{V}$ such that there exists a saddle

Figures (1)

  • Figure 1: An example MDP with spurious saddle points.

Theorems & Definitions (44)

  • Proposition 1
  • Lemma 2
  • Theorem 3
  • Proposition 4
  • Lemma 5
  • Theorem 6
  • Definition 7: No-Regret Oracle
  • Theorem 8
  • Lemma 9
  • Theorem 10
  • ...and 34 more