Average-Reward Maximum Entropy Reinforcement Learning for Global Policy in Double Pendulum Tasks

TL;DR

The paper tackles robust global swing-up and stabilization for underactuated double pendulums (acrobot and pendubot) under the updated 3rd AI Olympics at ICRA 2025. It extends Average-Reward Entropy Advantage Policy Optimisation (AR-EAPO) by modifying the MDP to emphasize fast swing-up, including a 4-D state representation, a 1-D torque input, a quadratic goal-oriented reward, and a small truncation horizon. Through extensive simulations, AR-EAPO achieves superior robustness and performance across five random seeds and outperforms several baselines, demonstrating stability under strong disturbances. The findings highlight the practicality of entropy-regularized, average-reward reinforcement learning for global control of underactuated robotic systems in noisy, uncertain environments.

Abstract

This report presents our reinforcement learning-based approach for the swing-up and stabilisation tasks of the acrobot and pendubot, tailored specifcially to the updated guidelines of the 3rd AI Olympics at ICRA 2025. Building upon our previously developed Average-Reward Entropy Advantage Policy Optimization (AR-EAPO) algorithm, we refined our solution to effectively address the new competition scenarios and evaluation metrics. Extensive simulations validate that our controller robustly manages these revised tasks, demonstrating adaptability and effectiveness within the updated framework.

Figures (2)

• Figure 1: Swing-up trajectory with AR-EAPO on the acrobot (the best-performing seed out of the five random seeds we evaluated).
• Figure 2: Swing-up trajectory with AR-EAPO on the pendubot (the best-performing seed out of the five random seeds we evaluated).