Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Object-Focus Actor for Data-efficient Robot Generalization Dexterous Manipulation

Yihang Li, Tianle Zhang, Xuelong Wei, Jiayi Li, Lin Zhao, Dongchi Huang, Zhirui Fang, Minhua Zheng, Wenjun Dai, Xiaodong He

TL;DR

This work tackles the challenge of generalizing dexterous manipulation to arbitrary object positions in the real world under data scarcity. It introduces the Object-Focus Actor (OFA), a hierarchical framework that combines object-centric perception and pose estimation, category-level pre-manipulation pose arrival, and a CVAE-based hand-focused policy that leverages relative proprioception and action chunks to learn a core, transferable manipulation trajectory. Across seven real-world tasks, OFA outperforms baselines in both positional and background generalization, and demonstrates remarkable data efficiency, achieving competitive performance with as few as 10 demonstrations. The approach offers practical impact for deploying dexterous manipulation in dynamic environments with limited task-specific data.

Abstract

Robot manipulation learning from human demonstrations offers a rapid means to acquire skills but often lacks generalization across diverse scenes and object placements. This limitation hinders real-world applications, particularly in complex tasks requiring dexterous manipulation. Vision-Language-Action (VLA) paradigm leverages large-scale data to enhance generalization. However, due to data scarcity, VLA's performance remains limited. In this work, we introduce Object-Focus Actor (OFA), a novel, data-efficient approach for generalized dexterous manipulation. OFA exploits the consistent end trajectories observed in dexterous manipulation tasks, allowing for efficient policy training. Our method employs a hierarchical pipeline: object perception and pose estimation, pre-manipulation pose arrival and OFA policy execution. This process ensures that the manipulation is focused and efficient, even in varied backgrounds and positional layout. Comprehensive real-world experiments across seven tasks demonstrate that OFA significantly outperforms baseline methods in both positional and background generalization tests. Notably, OFA achieves robust performance with only 10 demonstrations, highlighting its data efficiency.

Object-Focus Actor for Data-efficient Robot Generalization Dexterous Manipulation

TL;DR

This work tackles the challenge of generalizing dexterous manipulation to arbitrary object positions in the real world under data scarcity. It introduces the Object-Focus Actor (OFA), a hierarchical framework that combines object-centric perception and pose estimation, category-level pre-manipulation pose arrival, and a CVAE-based hand-focused policy that leverages relative proprioception and action chunks to learn a core, transferable manipulation trajectory. Across seven real-world tasks, OFA outperforms baselines in both positional and background generalization, and demonstrates remarkable data efficiency, achieving competitive performance with as few as 10 demonstrations. The approach offers practical impact for deploying dexterous manipulation in dynamic environments with limited task-specific data.

Abstract

Robot manipulation learning from human demonstrations offers a rapid means to acquire skills but often lacks generalization across diverse scenes and object placements. This limitation hinders real-world applications, particularly in complex tasks requiring dexterous manipulation. Vision-Language-Action (VLA) paradigm leverages large-scale data to enhance generalization. However, due to data scarcity, VLA's performance remains limited. In this work, we introduce Object-Focus Actor (OFA), a novel, data-efficient approach for generalized dexterous manipulation. OFA exploits the consistent end trajectories observed in dexterous manipulation tasks, allowing for efficient policy training. Our method employs a hierarchical pipeline: object perception and pose estimation, pre-manipulation pose arrival and OFA policy execution. This process ensures that the manipulation is focused and efficient, even in varied backgrounds and positional layout. Comprehensive real-world experiments across seven tasks demonstrate that OFA significantly outperforms baseline methods in both positional and background generalization tests. Notably, OFA achieves robust performance with only 10 demonstrations, highlighting its data efficiency.

Paper Structure

This paper contains 18 sections, 2 equations, 5 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overall Design of OFA
  5. Manipulating-Object Perception and Pose Estimation
  6. Pre-Manipulation Pose Arrival
  7. Object-Focus Policy Learning
  8. Experiments
  9. Experimental Setting
  10. Hardware
  11. Task Setting
  12. Data Collection
  13. Quantitative Experiment Results
  14. Generalization Analysis
  15. Sensitivity analysis of the quantity of demonstration data
  16. ...and 3 more sections

Figures (5)

  • Figure 1: The grasping trajectories of the dexterous robot hand. The four green points denote random initial relative position between the robot hand and the object to be grasped. The colored lines represent the trajectories. The yellow point illustrates the uniform pre-manipulation position and the red point is the final of the trajectories.
  • Figure 2: Overall design of OFA
  • Figure 3: The extraction of robot hand focus area. The left image and the right image are captured by the pair RGB stereo camera respectively. In each task, the hand focus area almost only includes the dexterous hand and the target object.
  • Figure 4: The bimanual robot used in our experiment. The mainly used modules for this paper are Realman RM65-B 6-DoF arms, Inspire RH56BFX dexterous hands and a ZED2 camera installed on the head.
  • Figure 5: Generalization testing of different methods.