Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients

Ashley Davey, Harry Zheng

TL;DR

This work advances proximal policy gradient methods for stochastic control problems where the control enters both the drift and diffusion terms. By representing the gradient through adjoint BSDE solutions via adjoint operators, the authors establish linear convergence under sufficiently large strong convexity in either the running or the terminal cost. They introduce two practical implementations: an explicit ODE-based scheme for unconstrained LQ problems and a deep learning-based PPGM for general constrained settings, with numerical results confirming convergence and scalability in higher dimensions. The findings extend PPGM applicability to control-dependent diffusion, enabling efficient solutions for high-dimensional stochastic control with nonconvex components and constraints.

Abstract

We prove convergence of the proximal policy gradient method for a class of constrained stochastic control problems with control in both the drift and diffusion of the state process. The problem requires either the running or terminal cost to be strongly convex, but other terms may be non-convex. The inclusion of control-dependent diffusion introduces additional complexity in regularity analysis of the associated backward stochastic differential equation. We provide sufficient conditions under which the control iterates converge linearly to the optimal control, by deriving representations and estimates of solutions to the adjoint backward stochastic differential equations. We introduce numerical algorithms that implement this method using deep learning and ordinary differential equation based techniques. These approaches enable high accuracy and scalability for stochastic control problems in higher dimensions. We provide numerical examples to demonstrate the accuracy and validate the theoretical convergence guarantees of the algorithms.

Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients

TL;DR

This work advances proximal policy gradient methods for stochastic control problems where the control enters both the drift and diffusion terms. By representing the gradient through adjoint BSDE solutions via adjoint operators, the authors establish linear convergence under sufficiently large strong convexity in either the running or the terminal cost. They introduce two practical implementations: an explicit ODE-based scheme for unconstrained LQ problems and a deep learning-based PPGM for general constrained settings, with numerical results confirming convergence and scalability in higher dimensions. The findings extend PPGM applicability to control-dependent diffusion, enabling efficient solutions for high-dimensional stochastic control with nonconvex components and constraints.

Abstract

We prove convergence of the proximal policy gradient method for a class of constrained stochastic control problems with control in both the drift and diffusion of the state process. The problem requires either the running or terminal cost to be strongly convex, but other terms may be non-convex. The inclusion of control-dependent diffusion introduces additional complexity in regularity analysis of the associated backward stochastic differential equation. We provide sufficient conditions under which the control iterates converge linearly to the optimal control, by deriving representations and estimates of solutions to the adjoint backward stochastic differential equations. We introduce numerical algorithms that implement this method using deep learning and ordinary differential equation based techniques. These approaches enable high accuracy and scalability for stochastic control problems in higher dimensions. We provide numerical examples to demonstrate the accuracy and validate the theoretical convergence guarantees of the algorithms.

Paper Structure

This paper contains 18 sections, 6 theorems, 80 equations, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notation
  3. Problem Formulation and PPGM
  4. Implementation
  5. Unconstrained LQ Problem
  6. General Algorithm
  7. Deep Learning for Solving BSDEs
  8. Deep Learning for Control Iteration
  9. Numerics
  10. LQ Problems
  11. Convex problem with standard assumptions
  12. Convex problem with singular assumptions
  13. Non-Convex problem with standard assumptions
  14. Cone constrained convex problem with standard assumptions
  15. Sensitivity Analysis
  16. ...and 3 more sections

Key Result

Theorem 2.1

Let $\left(X, \|\cdot\|_X\right)$ be a Hilbert space, $F \colon X \to \mathbb{R}$ be a differentiable function and $\tilde{F} \colon X \to \mathbb{R} \cup \{\infty\}$ be a proper, lower semicontinuous and convex function. Let $x^* \in X$ such that $\tilde{F}\left(x^*\right) < \infty$. Then $x^*$ is if and only if there exists $\bar{\tau} > 0$ such that for all $\tau \in (0, \bar{\tau})$ where $\t

Figures (6)

  • Figure 1: Comparison between algorithm errors for the unconstrained LQ problem.
  • Figure 2: Comparison between algorithm errors for the unconstrained LQ problem.
  • Figure 3: Comparison between algorithm errors for the unconstrained LQ problem.
  • Figure 4: Comparison between algorithm errors for the constrained LQ problem.
  • Figure 5: Control and value errors and runtime of unconstrained PGM applied to LQ problem.
  • ...and 1 more figures

Theorems & Definitions (17)

  • Theorem 2.1: Stationary point characterisation, reisinger2023linear Theorem 3.10
  • Definition 2.2: Proximal policy gradient method
  • Remark 1
  • Theorem 2.3
  • Remark 2
  • Theorem 2.4
  • Theorem 3.1
  • Example 4.1
  • Example 4.2
  • Example 4.3
  • ...and 7 more