Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States

Kyoleen Kwak, Hyoseok Hwang

TL;DR

The paper tackles high-frequency action oscillations in DRL control by introducing ASAP, a loss-based action smoothing method that defines similar states from the transition distribution $\text{sim}(s_t)=P(\cdot|s_{t-1})$ and couples a spatial loss with a temporal second-order penalty. A composite Lipschitz constraint on the actor via the transition function $T$ yields a principled, data-driven loss $L$ that aligns current actions with predictions from preceding states, while a predictor head enables the $L_P$ term to guide these predictions. The ASAP objective $J^{\text{ASAP}}_{\pi_\phi} = J_{\pi_\phi} + \lambda_S L_S + \lambda_P L_P + \lambda_T L_T$ balances smoothness and performance, demonstrated across Gymnasium and Isaac-Lab with PPO and SAC against baselines like CAPS, L2C2, and GRAD. Results show ASAP achieves substantially smoother actions (lower $Sm$) while preserving or improving cumulative returns, and it remains compatible with architectural approaches, suggesting strong practical applicability for real-world robotics and controlled environments.

Abstract

Deep reinforcement learning has proven to be a powerful approach to solving control tasks, but its characteristic high-frequency oscillations make it difficult to apply in real-world environments. While prior methods have addressed action oscillations via architectural or loss-based methods, the latter typically depend on heuristic or synthetic definitions of state similarity to promote action consistency, which often fail to accurately reflect the underlying system dynamics. In this paper, we propose a novel loss-based method by introducing a transition-induced similar state. The transition-induced similar state is defined as the distribution of next states transitioned from the previous state. Since it utilizes only environmental feedback and actually collected data, it better captures system dynamics. Building upon this foundation, we introduce Action Smoothing by Aligning Actions with Predictions from Preceding States (ASAP), an action smoothing method that effectively mitigates action oscillations. ASAP enforces action smoothness by aligning the actions with those taken in transition-induced similar states and by penalizing second-order differences to suppress high-frequency oscillations. Experiments in Gymnasium and Isaac-Lab environments demonstrate that ASAP yields smoother control and improved policy performance over existing methods.

Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States

TL;DR

The paper tackles high-frequency action oscillations in DRL control by introducing ASAP, a loss-based action smoothing method that defines similar states from the transition distribution and couples a spatial loss with a temporal second-order penalty. A composite Lipschitz constraint on the actor via the transition function yields a principled, data-driven loss that aligns current actions with predictions from preceding states, while a predictor head enables the term to guide these predictions. The ASAP objective balances smoothness and performance, demonstrated across Gymnasium and Isaac-Lab with PPO and SAC against baselines like CAPS, L2C2, and GRAD. Results show ASAP achieves substantially smoother actions (lower ) while preserving or improving cumulative returns, and it remains compatible with architectural approaches, suggesting strong practical applicability for real-world robotics and controlled environments.

Abstract

Deep reinforcement learning has proven to be a powerful approach to solving control tasks, but its characteristic high-frequency oscillations make it difficult to apply in real-world environments. While prior methods have addressed action oscillations via architectural or loss-based methods, the latter typically depend on heuristic or synthetic definitions of state similarity to promote action consistency, which often fail to accurately reflect the underlying system dynamics. In this paper, we propose a novel loss-based method by introducing a transition-induced similar state. The transition-induced similar state is defined as the distribution of next states transitioned from the previous state. Since it utilizes only environmental feedback and actually collected data, it better captures system dynamics. Building upon this foundation, we introduce Action Smoothing by Aligning Actions with Predictions from Preceding States (ASAP), an action smoothing method that effectively mitigates action oscillations. ASAP enforces action smoothness by aligning the actions with those taken in transition-induced similar states and by penalizing second-order differences to suppress high-frequency oscillations. Experiments in Gymnasium and Isaac-Lab environments demonstrate that ASAP yields smoother control and improved policy performance over existing methods.
Paper Structure (31 sections, 2 theorems, 17 equations, 3 figures, 7 tables)

This paper contains 31 sections, 2 theorems, 17 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Architectural Methods.
  4. Loss Penalty Methods.
  5. Background
  6. Lipschitz Continuity
  7. Lipschitz Constraints in Reinforcement Learning
  8. Temporal Smoothness.
  9. Spatial Smoothness.
  10. Method
  11. Overview
  12. Similar State Based on Transition Distribution
  13. Composite Function Based Lipschitz Constraint
  14. Action Smoothing via Predictions from Preceding States
  15. Training Method.
  16. ...and 16 more sections

Key Result

Lemma 1

Under Assumptions ass:transition_lips–ass:bounded_noise, any two states $s_{t}^{(1)},\; s_{t}^{(2)}\sim\mathrm{sim}(s_{t})$ satisfy

Figures (3)

  • Figure 1: Definition of similar states and action update mechanisms for (a) CAPS, (b) L2C2, and (c) ASAP. (a) CAPS samples states from a Gaussian around the current state and enforces action consistency between real and sampled states. (b) L2C2 constructs similar states by projecting the difference between the next state and the current state in multiple directions and minimizes divergence between the similar and actual policy distributions. (c) ASAP uses the environment's transition distribution to define similar states and aligns the action with the expected policy output under this distribution.
  • Figure 2: (a) The update procedure of ASAP and (b) The implementation architecture of ASAP.
  • Figure 3: Action change ($\Delta$a) of Joint-1 in the Franka reach task for original PPO and ASAP. Solid line represents median $\Delta$a run; shaded area indicates mean across all seeds.

Theorems & Definitions (6)

  • Definition 1: Global Lipschitz Continuity
  • Definition 2: Local Lipschitz Continuity
  • Definition 3: Similar State Distribution
  • Lemma 1: Spatially Bounded Neighborhood
  • proof
  • Theorem 1: Composite Lipschitz Constraint