Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control

Chengxiu Hua, Jiawen Gu, Yushun Tang

TL;DR

This work develops Continuous Q-Score Matching (CQSM), a continuous-time reinforcement learning framework where state-action dynamics follow stochastic differential equations and the Q-function $Q(t,x,a)$ is learned via a martingale-based characterization. By deriving a dynamic programming/HJB formulation and a score-improvement theorem, CQSM couples the Q-function with the score of a diffusion policy, enabling model-free, actor-critic–style policy updates that align the policy score with the action gradient of $Q$. The approach yields analytic validation in linear-quadratic stochastic control and demonstrates early-training advantages over continuous-time policy gradient and little q-learning on LQ tasks, with competitive performance on broader continuous-control benchmarks. Overall, CQSM preserves action-aware Q-learning in continuous time without time discretization, offering principled policy improvement and potential applications in finance, robotics, and other domains with fine-grained temporal decision-making.

Abstract

Reinforcement learning (RL) has achieved significant success across a wide range of domains, however, most existing methods are formulated in discrete time. In this work, we introduce a novel RL method for continuous-time control, where stochastic differential equations govern state-action dynamics. Departing from traditional value function-based approaches, our key contribution is the characterization of continuous-time Q-functions via a martingale condition and the linking of diffusion policy scores to the action gradient of a learned continuous Q-function by the dynamic programming principle. This insight motivates Continuous Q-Score Matching (CQSM), a score-based policy improvement algorithm. Notably, our method addresses a long-standing challenge in continuous-time RL: preserving the action-evaluation capability of Q-functions without relying on time discretization. We further provide theoretical closed-form solutions for linear-quadratic (LQ) control problems within our framework. Numerical results in simulated environments demonstrate the effectiveness of our proposed method and compare it to popular baselines.

Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control

TL;DR

This work develops Continuous Q-Score Matching (CQSM), a continuous-time reinforcement learning framework where state-action dynamics follow stochastic differential equations and the Q-function is learned via a martingale-based characterization. By deriving a dynamic programming/HJB formulation and a score-improvement theorem, CQSM couples the Q-function with the score of a diffusion policy, enabling model-free, actor-critic–style policy updates that align the policy score with the action gradient of . The approach yields analytic validation in linear-quadratic stochastic control and demonstrates early-training advantages over continuous-time policy gradient and little q-learning on LQ tasks, with competitive performance on broader continuous-control benchmarks. Overall, CQSM preserves action-aware Q-learning in continuous time without time discretization, offering principled policy improvement and potential applications in finance, robotics, and other domains with fine-grained temporal decision-making.

Abstract

Reinforcement learning (RL) has achieved significant success across a wide range of domains, however, most existing methods are formulated in discrete time. In this work, we introduce a novel RL method for continuous-time control, where stochastic differential equations govern state-action dynamics. Departing from traditional value function-based approaches, our key contribution is the characterization of continuous-time Q-functions via a martingale condition and the linking of diffusion policy scores to the action gradient of a learned continuous Q-function by the dynamic programming principle. This insight motivates Continuous Q-Score Matching (CQSM), a score-based policy improvement algorithm. Notably, our method addresses a long-standing challenge in continuous-time RL: preserving the action-evaluation capability of Q-functions without relying on time discretization. We further provide theoretical closed-form solutions for linear-quadratic (LQ) control problems within our framework. Numerical results in simulated environments demonstrate the effectiveness of our proposed method and compare it to popular baselines.
Paper Structure (32 sections, 4 theorems, 67 equations, 5 figures, 2 algorithms)

This paper contains 32 sections, 4 theorems, 67 equations, 5 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Major contributions.
  3. Related Works
  4. Stochastic Optimal Control.
  5. Continuous Reinforcement Learning.
  6. Diffusion Q-Learning.
  7. Behavior Cloning.
  8. Formulation and Preliminaries
  9. Notation.
  10. Continuous RL.
  11. Continuous Q-Score Matching Algorithm
  12. Dynamic Programming and HJB Equation for Q-Function
  13. Score Evaluation of the Q-Function
  14. Policy Optimization via Matching the Score to the Q-function
  15. Policy sampling.
  16. ...and 17 more sections

Key Result

Theorem 1

If $Q(\cdot, \cdot, \cdot; \Psi)$ is the Q-function associated with the score $\Psi$ if and only if it satisfies terminal condition $Q(T, x, a; \Psi)=h(x,a)$, and for all $(x,a)\in \mathbb{R}^n \times \mathbb{R}^d$, the following process is a $(\{\mathcal{F}_s\}_{s\geq 0}, \mathbb{P})$-martingale on $[t,T]$. Conversely, if there is a continuous Q-function $\tilde{Q}$ such that for all $(x,a)\in \

Figures (5)

  • Figure 1: The left image shows a randomly initialized score function $\Psi^0$, and the right shows the updated score $\Psi^1$ after one step. If a discrepancy exists between the score $\Psi$ (green vector) and the action gradient $\nabla_aQ(x,a)$ (blue vector), then aligning $\Psi$ with $\nabla_a Q$ yields a strict improvement in Q-value at $(x, a)$.
  • Figure 2: Paths of learned parameters of the CQSM reinforcement learning algorithm described in Algorithm \ref{['CQSMal']}. A single state trajectory of length $T = 10^5$ is generated. The dashed lines indicate the optimal parameter values. The shaded regions represent the standard deviation of the learned parameters across these runs, with the width of each shaded area equal to twice the corresponding standard deviation.
  • Figure 3: Running average rewards of three RL algorithms. A single state trajectory of length $T = 10^5$ is generated and discretized using three different step sizes: $\Delta t=0.01$ in panel (a), $\Delta t = 0.1$ in panel (b), $\Delta t = 1$ in panel (c). For each setting, we apply three online algorithms: Policy Gradient described in Algorithm 3 in jia2022policygradient, q-Learning described in Algorithm 4 in jia2023qlearning and CQSM described in Algorithm \ref{['CQSMal']}. We plot the mean running average reward over time and the shaded areas represent the standard deviation across the runs.
  • Figure F.1: Running average rewards of three RL algorithms. A single state trajectory of length $T = 10^5$ is generated and discretized using two different step sizes: $\Delta t=0.1$ in panel (a), $\Delta t = 1$ in panel (b).
  • Figure F.2: Experimental Results Across A Suite of Six Continuous Control Tasks.

Theorems & Definitions (6)

  • Definition 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Lemma 1
  • proof