Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic

Jeong Woon Lee, Kyoleen Kwak, Daeho Kim, Hyoseok Hwang

TL;DR

This work identifies policy instability in continuous actor-critic methods as rooted in the geometry of the critic, not merely in the actor. It derives a principled link between policy sensitivity and the Q-function's mixed partial and curvature terms using implicit differentiation, showing that the Lipschitz constant of the greedy policy is bounded by $L\le M/\mu$ with $M=\|\nabla^2_{sa}Q\|$ and $\mu=-\lambda_{\max}(\nabla^2_{aa}Q)$. To enforce this geometry, it introduces PAVE, a critic-centric regularization framework comprising Mixed-Partial Regularization, Vector Field Consistency, and Curvature Preservation, which stabilize the induced Q-gradient field while preserving curvature. Empirical results on six Gymnasium/MuJoCo tasks demonstrate that PAVE achieves smooth, robust policies with competitive task performance, particularly in high-dimensional domains, and maintains linear computational complexity. The findings advocate a shift toward critic geometry stabilization as a practical and scalable path to smooth policies in continuous control.

Abstract

Policies learned via continuous actor-critic methods often exhibit erratic, high-frequency oscillations, making them unsuitable for physical deployment. Current approaches attempt to enforce smoothness by directly regularizing the policy's output. We argue that this approach treats the symptom rather than the cause. In this work, we theoretically establish that policy non-smoothness is fundamentally governed by the differential geometry of the critic. By applying implicit differentiation to the actor-critic objective, we prove that the sensitivity of the optimal policy is bounded by the ratio of the Q-function's mixed-partial derivative (noise sensitivity) to its action-space curvature (signal distinctness). To empirically validate this theoretical insight, we introduce PAVE (Policy-Aware Value-field Equalization), a critic-centric regularization framework that treats the critic as a scalar field and stabilizes its induced action-gradient field. PAVE rectifies the learning signal by minimizing the Q-gradient volatility while preserving local curvature. Experimental results demonstrate that PAVE achieves smoothness and robustness comparable to policy-side smoothness regularization methods, while maintaining competitive task performance, without modifying the actor.

Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic

TL;DR

This work identifies policy instability in continuous actor-critic methods as rooted in the geometry of the critic, not merely in the actor. It derives a principled link between policy sensitivity and the Q-function's mixed partial and curvature terms using implicit differentiation, showing that the Lipschitz constant of the greedy policy is bounded by with and . To enforce this geometry, it introduces PAVE, a critic-centric regularization framework comprising Mixed-Partial Regularization, Vector Field Consistency, and Curvature Preservation, which stabilize the induced Q-gradient field while preserving curvature. Empirical results on six Gymnasium/MuJoCo tasks demonstrate that PAVE achieves smooth, robust policies with competitive task performance, particularly in high-dimensional domains, and maintains linear computational complexity. The findings advocate a shift toward critic geometry stabilization as a practical and scalable path to smooth policies in continuous control.

Abstract

Policies learned via continuous actor-critic methods often exhibit erratic, high-frequency oscillations, making them unsuitable for physical deployment. Current approaches attempt to enforce smoothness by directly regularizing the policy's output. We argue that this approach treats the symptom rather than the cause. In this work, we theoretically establish that policy non-smoothness is fundamentally governed by the differential geometry of the critic. By applying implicit differentiation to the actor-critic objective, we prove that the sensitivity of the optimal policy is bounded by the ratio of the Q-function's mixed-partial derivative (noise sensitivity) to its action-space curvature (signal distinctness). To empirically validate this theoretical insight, we introduce PAVE (Policy-Aware Value-field Equalization), a critic-centric regularization framework that treats the critic as a scalar field and stabilizes its induced action-gradient field. PAVE rectifies the learning signal by minimizing the Q-gradient volatility while preserving local curvature. Experimental results demonstrate that PAVE achieves smoothness and robustness comparable to policy-side smoothness regularization methods, while maintaining competitive task performance, without modifying the actor.
Paper Structure (36 sections, 2 theorems, 17 equations, 6 figures, 8 tables, 1 algorithm)

This paper contains 36 sections, 2 theorems, 17 equations, 6 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Policy-side smoothness regularization
  4. Critic geometry and gradient regularization
  5. Score matching and Fisher divergence
  6. Preliminary
  7. Theoretical Analysis: The Geometry of Policy Sensitivity
  8. Implicit Policy Definition and Sensitivity Derivation
  9. Spectral Bounds on Lipschitz Continuity
  10. PAVE: Policy-Aware Value-field Equalization
  11. Mixed-Partial Regularization (MPR)
  12. Formal Justification.
  13. Vector Field Consistency (VFC)
  14. Formal Justification.
  15. Curvature Preservation (Curv)
  16. ...and 21 more sections

Key Result

Lemma 4.1

Let $Q_\theta: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ be a twice continuously differentiable function. Assume that for a given state $s$, $a^*(s)$ corresponds to a strict local maximum and is an interior point of $\mathcal{A}$, such that the action Hessian $\nabla^2_{aa} Q_\theta(s,a^*(s))$

Figures (6)

  • Figure 1: Comprehensive 3D visualization of the mixed-partial Hessian norm $\|\nabla_{sa}^2 Q\|$ in LunarLander. While baseline methods exhibit highly irregular landscapes with sharp spikes, PAVE effectively stabilizes the Q-gradient field, providing a smooth and stable landscape.
  • Figure 2: Comprehensive 3D visualization of the mixed-partial Hessian norm $\|\nabla_{sa}^2 Q\|$ across six Gymnasium environments. Each row corresponds to an environment, and each column represents a different stabilization method. While baseline methods exhibit highly irregular landscapes with sharp spikes (indicating an unstable learning signal), PAVE effectively paves the Q-gradient field, providing a smooth and stable landscape. The Z-axis is clipped at 300 for visual clarity and consistent comparison.
  • Figure 3: Comprehensive 3D visualization of the mixed-partial Hessian norm $\|\nabla_{sa}^2 Q\|$ across six Gymnasium environments. Each row corresponds to an environment, and each column represents a different stabilization method. While baseline methods exhibit highly irregular landscapes with sharp spikes (indicating an unstable learning signal), PAVE effectively paves the Q-gradient field, providing a smooth and stable landscape. The Z-axis is clipped at 300 for visual clarity and consistent comparison.
  • Figure 4: Hyperparameter Sensitivity on Pendulum. For each subplot, only the target parameter was varied while the other two were held fixed at their default settings.
  • Figure 5: Learning curves of the TD3 algorithm across various Gymnasium environments.
  • ...and 1 more figures

Theorems & Definitions (4)

  • Lemma 4.1: Implicit Policy Jacobian
  • proof
  • Proposition 4.2: Lipschitz Continuity Bound
  • proof