Learning Long-Horizon Robot Manipulation Skills via Privileged Action

TL;DR

This work tackles the difficulty of learning long-horizon, contact-rich robotic manipulation under sparse rewards by introducing a structured framework of privileged actions in simulation combined with curriculum learning. The method progressively relaxes physical constraints (constraint relaxation), augments interaction with virtual forces, and then gradually reduces privileges to converge to real-world-feasible behaviors, all without task-specific reward shaping or demonstrations. Empirical results in simulation and real-world transfer demonstrate robust, diverse manipulation skills—including non-prehensile pushing and pivot grasps, as well as dexterous object handling—that outperform state-of-the-art baselines. The approach shows strong generalization across object types and platforms, and confirms that policies learned with privileged actions can transfer to real environments with domain randomization and careful control, highlighting practical impact for scalable robotic learning.

Abstract

Long-horizon contact-rich tasks are challenging to learn with reinforcement learning, due to ineffective exploration of high-dimensional state spaces with sparse rewards. The learning process often gets stuck in local optimum and demands task-specific reward fine-tuning for complex scenarios. In this work, we propose a structured framework that leverages privileged actions with curriculum learning, enabling the policy to efficiently acquire long-horizon skills without relying on extensive reward engineering or reference trajectories. Specifically, we use privileged actions in simulation with a general training procedure that would be infeasible to implement in real-world scenarios. These privileges include relaxed constraints and virtual forces that enhance interaction and exploration with objects. Our results successfully achieve complex multi-stage long-horizon tasks that naturally combine non-prehensile manipulation with grasping to lift objects from non-graspable poses. We demonstrate generality by maintaining a parsimonious reward structure and showing convergence to diverse and robust behaviors across various environments. Additionally, real-world experiments further confirm that the skills acquired using our approach are transferable to real-world environments, exhibiting robust and intricate performance. Our approach outperforms state-of-the-art methods in these tasks, converging to solutions where others fail.

Figures (6)

• Figure 1: Long-horizon manipulation incorporate distinct non-prehensile skills: push-and-grasp (upper) and pivot grasp (blow). These skills are learned across diverse environmental settings with our proposed framework while employing same reward function without any task-specific reward engineering.
• Figure 2: A structured framework utilizes privileged actions with curriculum learning. In stage 1, the robot penetrates the white table while a grey virtual table limits this penetration and is gradually lifted during training; stage 2 applies virtual forces on the object, indicated by blue and green arrows, and by stage 3, no privileged actions is used.
• Figure 3: Franka lift cube from a non-graspable pose in a constrained environment, where small walls placed at the table edge block direct pushing behavior. The policy learns to use the base of the robot as support to execute a pivot grasp.
• Figure 4: Using our framework and despite the absence of a specific reward indication for non-prehensile manipulation skills, the robot learns to grasp and lift the scissors by first pushing them to the edge of the table.
• Figure 5: Reward curves comparing our method with DexPBT and SAPG on three challenging objects indicate that our framework, represented by the blue lane, performs well on all objects. In contrast, DexPBT converge to a low reward value for the relatively large object, the stapler, and fail on the other two objects. The red line indicate when the stage changes.