Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysis of Control Bellman Residual Minimization for Markov Decision Problem

Donghwan Lee, Hyukjun Yang

TL;DR

This work investigates policy-optimization via Bellman-residual minimization by formulating a control Bellman residual (CBR) objective with linear function approximation and a differentiable soft variant (SCBR) using the soft Bellman operator $F_\lambda$. It establishes that CBR is nonconvex yet locally Lipschitz with a piecewise-quadratic structure, derives the Clarke subdifferential, and proves descent methods converge to Clark stationary points, with the tabular case yielding a unique solution $Q^*= \arg\max_a Q^*(s,a)$. The SCBR framework provides a differentiable objective with a gradient that leads to guaranteed convergence, including exponential convergence in the tabular setting under a PL condition, and it retains error-bounds comparable to CBR, plus local strong convexity near the soft-Bellman solution. An oblique-projection perspective (OP-CBE/OP-SCBE) clarifies the relation between Bellman residual minimization and projected Bellman equations in policy optimization. Empirical results in GridWorld/FrozenLake settings illustrate stability advantages of SCBR over projected value iteration and hint at practical potential and future work for deep RL extensions.

Abstract

Markov decision problems are most commonly solved via dynamic programming. Another approach is Bellman residual minimization, which directly minimizes the squared Bellman residual objective function. However, compared to dynamic programming, this approach has received relatively less attention, mainly because it is often less efficient in practice and can be more difficult to extend to model-free settings such as reinforcement learning. Nonetheless, Bellman residual minimization has several advantages that make it worth investigating, such as more stable convergence with function approximation for value functions. While Bellman residual methods for policy evaluation have been widely studied, methods for policy optimization (control tasks) have been scarcely explored. In this paper, we establish foundational results for the control Bellman residual minimization for policy optimization.

Analysis of Control Bellman Residual Minimization for Markov Decision Problem

TL;DR

This work investigates policy-optimization via Bellman-residual minimization by formulating a control Bellman residual (CBR) objective with linear function approximation and a differentiable soft variant (SCBR) using the soft Bellman operator . It establishes that CBR is nonconvex yet locally Lipschitz with a piecewise-quadratic structure, derives the Clarke subdifferential, and proves descent methods converge to Clark stationary points, with the tabular case yielding a unique solution . The SCBR framework provides a differentiable objective with a gradient that leads to guaranteed convergence, including exponential convergence in the tabular setting under a PL condition, and it retains error-bounds comparable to CBR, plus local strong convexity near the soft-Bellman solution. An oblique-projection perspective (OP-CBE/OP-SCBE) clarifies the relation between Bellman residual minimization and projected Bellman equations in policy optimization. Empirical results in GridWorld/FrozenLake settings illustrate stability advantages of SCBR over projected value iteration and hint at practical potential and future work for deep RL extensions.

Abstract

Markov decision problems are most commonly solved via dynamic programming. Another approach is Bellman residual minimization, which directly minimizes the squared Bellman residual objective function. However, compared to dynamic programming, this approach has received relatively less attention, mainly because it is often less efficient in practice and can be more difficult to extend to model-free settings such as reinforcement learning. Nonetheless, Bellman residual minimization has several advantages that make it worth investigating, such as more stable convergence with function approximation for value functions. While Bellman residual methods for policy evaluation have been widely studied, methods for policy optimization (control tasks) have been scarcely explored. In this paper, we establish foundational results for the control Bellman residual minimization for policy optimization.
Paper Structure (39 sections, 86 theorems, 291 equations, 14 figures, 1 algorithm)

This paper contains 39 sections, 86 theorems, 291 equations, 14 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. Markov decision problem
  5. Definitions and notations
  6. Control Bellman residual (CBR)
  7. Subdifferential
  8. Generalized gradient descent method
  9. Tabular case
  10. Error bounds
  11. Soft control Bellman residual (SCBR)
  12. Gradient
  13. Gradient descent method
  14. Tabular case
  15. Error bounds
  16. ...and 24 more sections

Key Result

Proposition 1

$f$ is piecewise quadratic (so piecewise smooth) and continuous with the polyhedral partition $S_\pi$ of $\mathbb{R}^m$ defined as for each $\pi \in \Theta$, where $\Theta$ is the set of all deterministic policies, and $\mathop{\mathrm{arg\,max}}\nolimits$ is interpreted as a set-valued map, i.e., $\pi(s)$ is one of the maximizers, and ties are allowed.

Figures (14)

  • Figure 1: Illustration of the oblique projection
  • Figure 2: Comparison of errors between the gradient descent on the SCBR (blue) and the P-VI (red). While the P-VI diverges, the gradient approach exhibits a steady decrease in error throughout the iterations.
  • Figure 3: The surface of $f(Q)$
  • Figure 4: Convergence trajectory of $Q_k$ with the subgradient descent method.
  • Figure 5: CBR objective function value $f(Q_k)$ with $Q_k$ generated with the subgradient descent method.
  • ...and 9 more figures

Theorems & Definitions (146)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Proposition 4
  • Proposition 5
  • ...and 136 more