Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoupling Task and Behavior: A Two-Stage Reward Curriculum in Reinforcement Learning for Robotics

Kilian Freitag, Knut Åkesson, Morteza Haghir Chehreghani

TL;DR

This work proposes a two-stage reward curriculum where the agent is first trained on a simplified task-only reward function to ensure effective exploration before introducing the full reward that includes auxiliary behavior-related terms such as energy efficiency.

Abstract

Deep Reinforcement Learning is a promising tool for robotic control, yet practical application is often hindered by the difficulty of designing effective reward functions. Real-world tasks typically require optimizing multiple objectives simultaneously, necessitating precise tuning of their weights to learn a policy with the desired characteristics. To address this, we propose a two-stage reward curriculum where we decouple task-specific objectives from behavioral terms. In our method, we first train the agent on a simplified task-only reward function to ensure effective exploration before introducing the full reward that includes auxiliary behavior-related terms such as energy efficiency. Further, we analyze various transition strategies and demonstrate that reusing samples between phases is critical for training stability. We validate our approach on the DeepMind Control Suite, ManiSkill3, and a mobile robot environment, modified to include auxiliary behavioral objectives. Our method proves to be simple yet effective, substantially outperforming baselines trained directly on the full reward while exhibiting higher robustness to specific reward weightings.

Decoupling Task and Behavior: A Two-Stage Reward Curriculum in Reinforcement Learning for Robotics

TL;DR

This work proposes a two-stage reward curriculum where the agent is first trained on a simplified task-only reward function to ensure effective exploration before introducing the full reward that includes auxiliary behavior-related terms such as energy efficiency.

Abstract

Deep Reinforcement Learning is a promising tool for robotic control, yet practical application is often hindered by the difficulty of designing effective reward functions. Real-world tasks typically require optimizing multiple objectives simultaneously, necessitating precise tuning of their weights to learn a policy with the desired characteristics. To address this, we propose a two-stage reward curriculum where we decouple task-specific objectives from behavioral terms. In our method, we first train the agent on a simplified task-only reward function to ensure effective exploration before introducing the full reward that includes auxiliary behavior-related terms such as energy efficiency. Further, we analyze various transition strategies and demonstrate that reusing samples between phases is critical for training stability. We validate our approach on the DeepMind Control Suite, ManiSkill3, and a mobile robot environment, modified to include auxiliary behavioral objectives. Our method proves to be simple yet effective, substantially outperforming baselines trained directly on the full reward while exhibiting higher robustness to specific reward weightings.
Paper Structure (31 sections, 29 equations, 9 figures, 1 table, 1 algorithm)

This paper contains 31 sections, 29 equations, 9 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Reward Curriculum Framework
  4. Phase Switch Mechanisms
  5. Actor fit threshold:
  6. Base reward threshold:
  7. Base reward convergence:
  8. Transition Dynamics and Annealing
  9. Reuse of past experience
  10. Experiments
  11. How effective is the Two-Stage Curriculum?
  12. How robust is the method to different $w_\text{target}$?
  13. Ablation Studies
  14. When should the second phase be initiated?
  15. How critical are the transition dynamics between phases?
  16. ...and 16 more sections

Figures (9)

  • Figure 1: Comparison of the baseline TD3 and SAC algorithms with reward curriculum versions (RC-TD3 and RC-SAC). The first row shows the achieved base reward (DM Control) or success rate (MobileRobot and ManiSkill3) and the second row shows the average reward. Both are computed as the average over the last 50k training steps and 3 random seed using target weight $w_\text{target}=0.5$ for DM Control, $w_\text{target}=0.25$ for ManiSkill3 and $w_\text{target}\in\{0.25,0.5,0.75\}$ for MobileRobot (MR).
  • Figure 2: Exemplary environments from DM control (walker-run), MobileRobot, and ManiSkill3 (PullCubeTool-v1).
  • Figure 3: Comparison of the mean success rate and average reward for ManiSkill3 (4 environments aggregated) and MobileRobot with different $w_\text{target}$ of RC-TD3 and RC-SAC to their baselines. The average reward is computed using a $w_\text{target}=0.5$ to make results comparable.
  • Figure 4: Comparison of the mean episode reward of RC-SAC with different switches as described in Section \ref{['sec:autoswitch']}. The values are average over the last 50 [k] training steps.
  • Figure 5: Comparison of the mean episode reward for different transitions as described in Section \ref{['sec:annealing']}. The values are average over the last 50 [k] training steps.
  • ...and 4 more figures