Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient Flows for Regularized Stochastic Control Problems

David Šiška, Łukasz Szpruch

TL;DR

The paper develops a gradient-flow framework for entropy-regularized stochastic control with measure-valued actions, linking the gradient flow on the space of probability measures to Pontryagin optimality and a Bayesian interpretation of the optimal control as a posterior relative to a prior. By deriving a mean-field forward–backward SDE representation and establishing conditions for existence, uniqueness, and exponential convergence to a unique invariant measure, it provides a rigorous foundation for convergence properties of stochastic gradient-type algorithms in high-dimensional control problems. The results integrate variational calculus on probability measures, BSDE techniques, and dissipativity arguments to show that the optimal measure-valued control can be characterized via a Gibbs-type form and achieved through gradient-flow dynamics. This framework offers theoretical backing for RL-style SGD approaches to stochastic control and highlights a principled way to incorporate prior knowledge while ensuring convergence guarantees.

Abstract

This paper studies stochastic control problems with the action space taken to be probability measures, with the objective penalised by the relative entropy. We identify suitable metric space on which we construct a gradient flow for the measure-valued control process, in the set of admissible controls, along which the cost functional is guaranteed to decrease. It is shown that any invariant measure of this gradient flow satisfies the Pontryagin optimality principle. If the problem we work with is sufficiently convex, the gradient flow converges exponentially fast. Furthermore, the optimal measure-valued control process admits a Bayesian interpretation which means that one can incorporate prior knowledge when solving such stochastic control problems. This work is motivated by a desire to extend the theoretical underpinning for the convergence of stochastic gradient type algorithms widely employed in the reinforcement learning community to solve control problems.

Gradient Flows for Regularized Stochastic Control Problems

TL;DR

The paper develops a gradient-flow framework for entropy-regularized stochastic control with measure-valued actions, linking the gradient flow on the space of probability measures to Pontryagin optimality and a Bayesian interpretation of the optimal control as a posterior relative to a prior. By deriving a mean-field forward–backward SDE representation and establishing conditions for existence, uniqueness, and exponential convergence to a unique invariant measure, it provides a rigorous foundation for convergence properties of stochastic gradient-type algorithms in high-dimensional control problems. The results integrate variational calculus on probability measures, BSDE techniques, and dissipativity arguments to show that the optimal measure-valued control can be characterized via a Gibbs-type form and achieved through gradient-flow dynamics. This framework offers theoretical backing for RL-style SGD approaches to stochastic control and highlights a principled way to incorporate prior knowledge while ensuring convergence guarantees.

Abstract

This paper studies stochastic control problems with the action space taken to be probability measures, with the objective penalised by the relative entropy. We identify suitable metric space on which we construct a gradient flow for the measure-valued control process, in the set of admissible controls, along which the cost functional is guaranteed to decrease. It is shown that any invariant measure of this gradient flow satisfies the Pontryagin optimality principle. If the problem we work with is sufficiently convex, the gradient flow converges exponentially fast. Furthermore, the optimal measure-valued control process admits a Bayesian interpretation which means that one can incorporate prior knowledge when solving such stochastic control problems. This work is motivated by a desire to extend the theoretical underpinning for the convergence of stochastic gradient type algorithms widely employed in the reinforcement learning community to solve control problems.

Paper Structure

This paper contains 12 sections, 26 theorems, 197 equations.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Heuristic derivation of the gradient flow
  4. Probabilistic representation
  5. Assumptions and Results
  6. Characterisation of the optimal control
  7. Existence and uniqueness for \ref{['eq mfsgd']}-\ref{['eq mfsgd 2']} and its convergence to invariant measure
  8. Pontryagin Optimality for Entropy-Regularized Stochastic Control
  9. Existence and Uniqueness of Solutions to the Mean-field system and exponential convergence to the invariant measure
  10. BSDE estimates
  11. Measure derivatives
  12. Sufficient condition for optimality

Key Result

Lemma 2.2

If $b$ is a permissible flow (cf. Definition def vect field flow def) then the linear PDE has unique solution $\nu_{\,\cdot,t} \in C^{1,\infty}((0,\infty)\times \mathbb R^p; \mathbb R)$ for each $t\in [0,T]$ and $\omega^W \in \Omega^W$. Moreover for each $s > 0$, $t\in [0,T]$ and $\omega^W \in \Omega^W$ we have $\nu_{s,t}(a) > 0$ and $\nu_{s,t}(a)$ is $\mathcal{F}_t^W$-measurable.

Theorems & Definitions (51)

  • Definition 2.1
  • Lemma 2.2
  • Lemma 2.4
  • Theorem 2.5
  • Theorem 2.6
  • Theorem 2.7: Necessary condition for optimality
  • Lemma 2.12: Existence and uniqueness
  • Theorem 2.13
  • Example 2.14
  • proof : Proof of Lemma \ref{['lem pde']}
  • ...and 41 more