MOSEAC: Streamlined Variable Time Step Reinforcement Learning
Dong Wang, Giovanni Beltrame
TL;DR
MOSEAC tackles the rigidity of fixed-time-step RL by introducing a Variable Time Step framework with adaptive reward shaping. By extending the action space to include action duration and using a multiplicative reward $R = α_m R_t R_τ - α_ε$ with $R_τ = t_{min} / t$, it reduces hyperparameter tuning to a single parameter $ψ$ that adapts based on reward trends. Empirical evaluation in a Newtonian kinematics environment demonstrates faster training and lower energy and time costs compared to SEAC and fixed-frequency baselines, highlighting improved data efficiency and practical deployment feasibility. The approach is designed to be broadly applicable to other continuous-action RL algorithms, offering a robust pathway to deploy VTS-RL in real-world systems such as robotics and autonomous vehicles.
Abstract
Traditional reinforcement learning (RL) methods typically employ a fixed control loop, where each cycle corresponds to an action. This rigidity poses challenges in practical applications, as the optimal control frequency is task-dependent. A suboptimal choice can lead to high computational demands and reduced exploration efficiency. Variable Time Step Reinforcement Learning (VTS-RL) addresses these issues by using adaptive frequencies for the control loop, executing actions only when necessary. This approach, rooted in reactive programming principles, reduces computational load and extends the action space by including action durations. However, VTS-RL's implementation is often complicated by the need to tune multiple hyperparameters that govern exploration in the multi-objective action-duration space (i.e., balancing task performance and number of time steps to achieve a goal). To overcome these challenges, we introduce the Multi-Objective Soft Elastic Actor-Critic (MOSEAC) method. This method features an adaptive reward scheme that adjusts hyperparameters based on observed trends in task rewards during training. This scheme reduces the complexity of hyperparameter tuning, requiring a single hyperparameter to guide exploration, thereby simplifying the learning process and lowering deployment costs. We validate the MOSEAC method through simulations in a Newtonian kinematics environment, demonstrating high task and training performance with fewer time steps, ultimately lowering energy consumption. This validation shows that MOSEAC streamlines RL algorithm deployment by automatically tuning the agent control loop frequency using a single parameter. Its principles can be applied to enhance any RL algorithm, making it a versatile solution for various applications.