Reinforcement Learning with Elastic Time Steps

TL;DR

This work tackles inefficiencies from fixed control rates in reinforcement learning by introducing MOSEAC, an off-policy actor-critic that leverages elastic time steps and a multiplicative reward to adapt control frequency. It adds online adjustment of $α_m$ with an upper bound $α_{max}$ and provides a convergence proof and Lyapunov stability analysis to guarantee stable learning. Empirically, MOSEAC outperforms CTCO, SEAC, and SAC (20 Hz) in TrackMania, delivering faster training, better energy efficiency, and the best track time of $43.202$ s in a real-time setting. The approach offers practical benefits for real-world robotics and sets the stage for extending variable time-step methods to hierarchical RL frameworks.

Abstract

Traditional Reinforcement Learning (RL) policies are typically implemented with fixed control rates, often disregarding the impact of control rate selection. This can lead to inefficiencies as the optimal control rate varies with task requirements. We propose the Multi-Objective Soft Elastic Actor-Critic (MOSEAC), an off-policy actor-critic algorithm that uses elastic time steps to dynamically adjust the control frequency. This approach minimizes computational resources by selecting the lowest viable frequency. We show that MOSEAC converges and produces stable policies at the theoretical level, and validate our findings in a real-time 3D racing game. MOSEAC significantly outperformed other variable time step approaches in terms of energy efficiency and task effectiveness. Additionally, MOSEAC demonstrated faster and more stable training, showcasing its potential for real-world RL applications in robotics.

Figures (9)

• Figure 1: Top preview of the Trackmania track.
• Figure 2: This is the implementation structure diagram of MOSEAC in the TrackMania environment. We use CNN to extract potential information in the environment and learn the extracted feature values based on rewards. The 143-dimensional state value and the 4-dimensional action value are shown in the figure. The time in the action value is not used for the current time step but for the next step.
• Figure 3: Our CNN structure diagram that we used to extract image features from the Trackmania video game. We convert RGB images into grayscale images and then input them into the CNN.
• Figure 4: Training Progress of MOSEAC and CTCO: Average Return Over Time
• Figure 5: Training Progress of MOSEAC and CTCO: Time Consumption Over Steps