Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reward Training Wheels: Adaptive Auxiliary Rewards for Robotics Reinforcement Learning

Linji Wang, Tong Xu, Yuanjie Lu, Xuesu Xiao

TL;DR

This paper tackles the challenge of hand-crafted auxiliary rewards in robotics RL by introducing Reward Training Wheels (RTW), a teacher–student framework that dynamically adapts auxiliary reward weights to match the learner’s evolving capabilities. The approach defines a decomposed reward with a primary objective and multiple auxiliary signals, whose weights are produced by a teacher trained to maximize the student’s performance. Empirical results in both simulated and physical robotics tasks (confined-space navigation and off-road mobility) show RTW outperforms expert-designed rewards, reward randomization, and MPPI planners, achieving higher success rates and faster training, while maintaining stability. The work demonstrates the practical impact of automated reward shaping in enabling more data-efficient and robust robotic learning, with future directions including discovering new reward components and more sophisticated teacher architectures.

Abstract

Robotics Reinforcement Learning (RL) often relies on carefully engineered auxiliary rewards to supplement sparse primary learning objectives to compensate for the lack of large-scale, real-world, trial-and-error data. While these auxiliary rewards accelerate learning, they require significant engineering effort, may introduce human biases, and cannot adapt to the robot's evolving capabilities during training. In this paper, we introduce Reward Training Wheels (RTW), a teacher-student framework that automates auxiliary reward adaptation for robotics RL. To be specific, the RTW teacher dynamically adjusts auxiliary reward weights based on the student's evolving capabilities to determine which auxiliary reward aspects require more or less emphasis to improve the primary objective. We demonstrate RTW on two challenging robot tasks: navigation in highly constrained spaces and off-road vehicle mobility on vertically challenging terrain. In simulation, RTW outperforms expert-designed rewards by 2.35% in navigation success rate and improves off-road mobility performance by 122.62%, while achieving 35% and 3X faster training efficiency, respectively. Physical robot experiments further validate RTW's effectiveness, achieving a perfect success rate (5/5 trials vs. 2/5 for expert-designed rewards) and improving vehicle stability with up to 47.4% reduction in orientation angles.

Reward Training Wheels: Adaptive Auxiliary Rewards for Robotics Reinforcement Learning

TL;DR

This paper tackles the challenge of hand-crafted auxiliary rewards in robotics RL by introducing Reward Training Wheels (RTW), a teacher–student framework that dynamically adapts auxiliary reward weights to match the learner’s evolving capabilities. The approach defines a decomposed reward with a primary objective and multiple auxiliary signals, whose weights are produced by a teacher trained to maximize the student’s performance. Empirical results in both simulated and physical robotics tasks (confined-space navigation and off-road mobility) show RTW outperforms expert-designed rewards, reward randomization, and MPPI planners, achieving higher success rates and faster training, while maintaining stability. The work demonstrates the practical impact of automated reward shaping in enabling more data-efficient and robust robotic learning, with future directions including discovering new reward components and more sophisticated teacher architectures.

Abstract

Robotics Reinforcement Learning (RL) often relies on carefully engineered auxiliary rewards to supplement sparse primary learning objectives to compensate for the lack of large-scale, real-world, trial-and-error data. While these auxiliary rewards accelerate learning, they require significant engineering effort, may introduce human biases, and cannot adapt to the robot's evolving capabilities during training. In this paper, we introduce Reward Training Wheels (RTW), a teacher-student framework that automates auxiliary reward adaptation for robotics RL. To be specific, the RTW teacher dynamically adjusts auxiliary reward weights based on the student's evolving capabilities to determine which auxiliary reward aspects require more or less emphasis to improve the primary objective. We demonstrate RTW on two challenging robot tasks: navigation in highly constrained spaces and off-road vehicle mobility on vertically challenging terrain. In simulation, RTW outperforms expert-designed rewards by 2.35% in navigation success rate and improves off-road mobility performance by 122.62%, while achieving 35% and 3X faster training efficiency, respectively. Physical robot experiments further validate RTW's effectiveness, achieving a perfect success rate (5/5 trials vs. 2/5 for expert-designed rewards) and improving vehicle stability with up to 47.4% reduction in orientation angles.

Paper Structure

This paper contains 25 sections, 6 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Curriculum Learning
  4. Reward Shaping
  5. Approach
  6. Student Agent
  7. Decomposed Reward Structure
  8. Teacher Agent
  9. Learning Process
  10. Student Learning
  11. Teacher Learning
  12. Training Algorithm
  13. Experiments and Results
  14. Simulation Experiments
  15. Experimental Setup
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Overview of Reward Training Wheel: (Left) The student agent interacts with the world and receives a reward composed of a primary component and weighted auxiliary components. (Right) The teacher agent maintains the history of previous weights, primary rewards, and auxiliary rewards as its state, and generates new weights as its action to optimize the student's learning process.
  • Figure 2: Learning curves showing success rate vs. training steps for MPPI, RR, ER, and RTW. Left: Confined-Space Navigation task. Right: Off-Road Vehicle Mobility task. RTW achieves higher succcess rate with fewer training steps in both tasks.
  • Figure 3: Simulation environments for the two robotics tasks: (Left) Confined-Space Navigation, requiring precise maneuvering through narrow corridors. (Right) Off-Road Vehicle Mobility, challenging the agent to traverse uneven terrain to reach a goal.
  • Figure 4: Evolution of auxiliary reward weights during training in Off-Road Vehicle Mobility.
  • Figure 5: Physical Off-Road Testbed for RTW.