Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies

Guilherme Christmann, Ying-Sheng Luo, Hanjaya Mandala, Wei-Chao Chen

TL;DR

This paper identifies, categorize, and compare methods from the literature that aim to mitigate high-frequency oscillations in deep RL, and proposes hybrid methods that combine elements from both loss regularization and architectural methods.

Abstract

Reinforcement learning (RL) policies are prone to high-frequency oscillations, especially undesirable when deploying to hardware in the real-world. In this paper, we identify, categorize, and compare methods from the literature that aim to mitigate high-frequency oscillations in deep RL. We define two broad classes: loss regularization and architectural methods. At their core, these methods incentivize learning a smooth mapping, such that nearby states in the input space produce nearby actions in the output space. We present benchmarks in terms of policy performance and control smoothness on traditional RL environments from the Gymnasium and a complex manipulation task, as well as three robotics locomotion tasks that include deployment and evaluation with real-world hardware. Finally, we also propose hybrid methods that combine elements from both loss regularization and architectural methods. We find that the best-performing hybrid outperforms other methods, and improves control smoothness by 26.8% over the baseline, with a worst-case performance degradation of just 2.8%.

Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies

TL;DR

This paper identifies, categorize, and compare methods from the literature that aim to mitigate high-frequency oscillations in deep RL, and proposes hybrid methods that combine elements from both loss regularization and architectural methods.

Abstract

Reinforcement learning (RL) policies are prone to high-frequency oscillations, especially undesirable when deploying to hardware in the real-world. In this paper, we identify, categorize, and compare methods from the literature that aim to mitigate high-frequency oscillations in deep RL. We define two broad classes: loss regularization and architectural methods. At their core, these methods incentivize learning a smooth mapping, such that nearby states in the input space produce nearby actions in the output space. We present benchmarks in terms of policy performance and control smoothness on traditional RL environments from the Gymnasium and a complex manipulation task, as well as three robotics locomotion tasks that include deployment and evaluation with real-world hardware. Finally, we also propose hybrid methods that combine elements from both loss regularization and architectural methods. We find that the best-performing hybrid outperforms other methods, and improves control smoothness by 26.8% over the baseline, with a worst-case performance degradation of just 2.8%.

Paper Structure

This paper contains 10 sections, 7 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works and Methods Categorization
  3. Loss Regularization Methods
  4. Architectural Methods
  5. Method and Experimental Setup
  6. Hybrid Methods
  7. Metrics
  8. Evaluation Scenarios
  9. Experiments and Results
  10. Conclusion

Figures (3)

  • Figure 1: We investigate the use of different classes of regularization to produce smooth control policies in several simulation and real-world environments.
  • Figure 2: Reward curves during training for 9 seeds. The hybrid methods LipsNet + CAPS and LipsNet + L2C2 show superior or comparable all environments, except in Lunar.
  • Figure 3: The methods are evaluated with the real-world robot. Every method achieves similar task performance (measured as cumulative return). The hybrid methods consistently outperformed other methods in regards to smoothness compared to the other methods.