Automated Hybrid Reward Scheduling via Large Language Models for Robotic Skill Learning

TL;DR

This work tackles the inefficiency of summing heterogeneous reward components in reinforcement learning for high-DOF robotic tasks. It proposes Automated Hybrid Reward Scheduling (AHRS), which uses Large Language Models to build a rule repository and dynamically adjust per-component learning weights within a multi-branch value network, complemented by an LLM-generated auxiliary reward. The approach achieves about 6.48% average improvements over PPO and 5.52% over HD-PPO across diverse robotics tasks in simulation, with ablations showing the value of LLm-driven rule selection over random or direct weight generation. The framework holds promise for more sample-efficient, adaptable robot skill learning and motivates sim-to-real validation in future work.

Abstract

Enabling a high-degree-of-freedom robot to learn specific skills is a challenging task due to the complexity of robotic dynamics. Reinforcement learning (RL) has emerged as a promising solution; however, addressing such problems requires the design of multiple reward functions to account for various constraints in robotic motion. Existing approaches typically sum all reward components indiscriminately to optimize the RL value function and policy. We argue that this uniform inclusion of all reward components in policy optimization is inefficient and limits the robot's learning performance. To address this, we propose an Automated Hybrid Reward Scheduling (AHRS) framework based on Large Language Models (LLMs). This paradigm dynamically adjusts the learning intensity of each reward component throughout the policy optimization process, enabling robots to acquire skills in a gradual and structured manner. Specifically, we design a multi-branch value network, where each branch corresponds to a distinct reward component. During policy optimization, each branch is assigned a weight that reflects its importance, and these weights are automatically computed based on rules designed by LLMs. The LLM generates a rule set in advance, derived from the task description, and during training, it selects a weight calculation rule from the library based on language prompts that evaluate the performance of each branch. Experimental results demonstrate that the AHRS method achieves an average 6.48% performance improvement across multiple high-degree-of-freedom robotic tasks.

Figures (6)

• Figure 1: Overview of the proposed Automated Hybrid Reward Scheduling (AHRS) framework. It includes multi-branch value networks, the construction of dynamic weight rule repository, the selection of rules, and the generation of auxiliary reward functions.
• Figure 2: Illustrations of the six tasks in this experiment: Ant, ShadowHand, AnymalTerrain, AllegroHand, Quadcopter, Cassie
• Figure 3: The learning curves show the variation in accumulative rewards across multiple environments (Ant, ShadowHand, AllegroHand, Quadcopter, AnymalTerrain, Cassie) over training epochs. Different colored lines represent different methods: PPO (red), HD-PPO (yellow), AHRS w/o A (cyan), and AHRS(blue). The solid lines represent the mean values for each method, while the shaded areas indicate the standard deviation.
• Figure 4: Examples of LLM-generated rules include: Formula 1, which uses logarithmic means and adjusted variances to smooth out extreme values, and Formula 2, which balances average return and variance, scaled by $\alpha$ and offset by $\beta$, to prioritize stable, high-performing components.
• Figure 5: Rule repository of AHRS.