Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Pontryagin maximum principle and $Q$-functions in rough environments

Estepan Ashkarian, Prakash Chakraborty, Harsha Honnappa, Samy Tindel

TL;DR

The paper addresses reinforcement learning in rough, noisy environments by deriving a Pontryagin maximum principle for relaxed controls and establishing a Q/q-function framework tailored to rough dynamics. The main innovation is a spike-variation differentiation approach that yields a PMP and an infinitesimal Q-function, linking control optimization to the Hamiltonian through the co-state and value function under rough paths. It also shows that entropy-regularized open-loop policies are Gibbs measures and develops a policy-improvement scheme using rough-flow transforms and a viscosity-comparison argument, enabling robust, path-dependent policy updates. Together, these results provide a rigorous, computable foundation for policy optimization in rough environments, with potential implications for offline RL and entropy-regularized control in non-Markovian settings.

Abstract

We derive the Pontryagin maximum principle and $Q$-functions for the relaxed control of noisy rough differential equations. Our main tool is the development of a novel differentiation procedure along `spike variation' perturbations of the optimal state-control pair. We then exploit our development of the infinitesimal $Q$-function (also known as the $q$-function) to derive a policy improvement algorithm for settings with entropic cost constraints.

The Pontryagin maximum principle and $Q$-functions in rough environments

TL;DR

The paper addresses reinforcement learning in rough, noisy environments by deriving a Pontryagin maximum principle for relaxed controls and establishing a Q/q-function framework tailored to rough dynamics. The main innovation is a spike-variation differentiation approach that yields a PMP and an infinitesimal Q-function, linking control optimization to the Hamiltonian through the co-state and value function under rough paths. It also shows that entropy-regularized open-loop policies are Gibbs measures and develops a policy-improvement scheme using rough-flow transforms and a viscosity-comparison argument, enabling robust, path-dependent policy updates. Together, these results provide a rigorous, computable foundation for policy optimization in rough environments, with potential implications for offline RL and entropy-regularized control in non-Markovian settings.

Abstract

We derive the Pontryagin maximum principle and -functions for the relaxed control of noisy rough differential equations. Our main tool is the development of a novel differentiation procedure along `spike variation' perturbations of the optimal state-control pair. We then exploit our development of the infinitesimal -function (also known as the -function) to derive a policy improvement algorithm for settings with entropic cost constraints.
Paper Structure (18 sections, 28 theorems, 191 equations)

This paper contains 18 sections, 28 theorems, 191 equations.

Table of Contents

  1. Introduction
  2. The rough viscosity setting
  3. Increments
  4. Weakly controlled processes
  5. Strongly controlled processes
  6. Rough differential equations with relaxed controls
  7. Value function and rough HJB equations
  8. Pontryagin Maximum Principle: Relaxed Control Setting
  9. Statement of Theorem and Strategy
  10. Differentiating along spike variations
  11. Identification of a derivative process
  12. Proof of Theorem \ref{['Pontryagin']}
  13. $Q$-Learning
  14. Definition of the $Q$-function
  15. Defining the $\mathbf{q}$-function through differentiation
  16. ...and 3 more sections

Key Result

Proposition 2.3

Let $h \in \mathcal{C}_3^{\mu}(V)$ for $\mu > 1$ be such that $\delta h = 0$. Then there exists a unique $g = \Lambda(h) \in \mathcal{C}_2^{\mu}(V)$ such that $\delta g = h$. Furthermore for such an $h$, the following relations hold true:

Theorems & Definitions (76)

  • Remark 1.1
  • Remark 2.2
  • Proposition 2.3
  • Proposition 2.4
  • Remark 2.6
  • Remark 2.7
  • Definition 2.8
  • Remark 2.9
  • Proposition 2.10
  • Proposition 2.12
  • ...and 66 more