Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning

TL;DR

The paper tackles jerky behaviors in deep reinforcement learning for robotic control by introducing Grad-CAPS, a gradient-based regularization that minimizes changes in action gradients while normalizing for displacement to handle different action scales. By shifting focus from penalizing action differences to differences in action changes, Grad-CAPS preserves policy expressiveness and reduces zigzagging without over-smoothing. Empirical results across toy tasks, the DeepMind Control Suite, and OpenAI Gym show Grad-CAPS generally improves performance while maintaining smooth trajectories, outperforming CAPS and vanilla baselines in many settings. This approach offers a practical, generalizable method for safer and more efficient DRL-driven robotics in diverse environments and algorithms.

Abstract

Deep Reinforcement Learning (DRL) has achieved remarkable success, ranging from complex computer games to real-world applications, showing the potential for intelligent agents capable of learning in dynamic environments. However, its application in real-world scenarios presents challenges, including the jerky problem, in which jerky trajectories not only compromise system safety but also increase power consumption and shorten the service life of robotic and autonomous systems. To address jerky actions, a method called conditioning for action policy smoothness (CAPS) was proposed by adding regularization terms to reduce the action changes. This paper further proposes a novel method, named Gradient-based CAPS (Grad-CAPS), that modifies CAPS by reducing the difference in the gradient of action and then uses displacement normalization to enable the agent to adapt to invariant action scales. Consequently, our method effectively reduces zigzagging action sequences while enhancing policy expressiveness and the adaptability of our method across diverse scenarios and environments. In the experiments, we integrated Grad-CAPS with different reinforcement learning algorithms and evaluated its performance on various robotic-related tasks in DeepMind Control Suite and OpenAI Gym environments. The results demonstrate that Grad-CAPS effectively improves performance while maintaining a comparable level of smoothness compared to CAPS and Vanilla agents.

Figures (5)

• Figure 1: Differences of temporal smoothness loss between CAPS and Grad-CAPS.
• Figure 2: Two cases: one with a zigzagging sequence of actions (left) and the other with a sequence with constant action changes (right). The upper part shows corresponding car racing scenarios, and the lower part shows corresponding losses for CAPS and Grad-CAPS. The cases show that CAPS fails to distinguish two sequences, while Grad-CAPS encourages stable action changes and penalizes zigzagging patterns.
• Figure 3: The referenced trajectories: (a) a square wave and (b) a cosine wave. The CAPS agent tends to over-smooth the action, leading to a loss of expressiveness in tracking the reference path. The Grad-CAPS agent performs better in following the reference path while maintaining smoothness.
• Figure 4: Temporal loss weight $\lambda_T$ ablation study on wave tracking experiments.
• Figure 5: Steering actions during a track of different agents in Car Racing environment. Grad-CAPS clearly obtains smoother steering action compared to other methods.

Theorems & Definitions (2)

• Definition II.1
• Definition III.1