Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulation

TL;DR

This work addresses the challenge of decomposing long-horizon robotic manipulation demonstrations into reusable sub-tasks for imitation learning. It proposes a gaze-guided framework that detects sub-task boundaries from transitions between gaze landmarks, using median-filtered gaze and CLIP-based visual features, along with a refinement step to standardize sub-task counts across demonstrations. The approach yields consistent segmentations across multiple demonstrations and tasks, and when integrated into an imitation-learning pipeline, improves performance on unseen positions and poses. The method is practical for real-world deployment, relying on gaze data from teleoperation rather than extensive additional training data, and is supported by open-source code.

Abstract

In imitation learning for robotic manipulation, decomposing object manipulation tasks into sub-tasks enables the reuse of learned skills and the combination of learned behaviors to perform novel tasks, rather than simply replicating demonstrated motions. Human gaze is closely linked to hand movements during object manipulation. We hypothesize that an imitating agent's gaze control, fixating on specific landmarks and transitioning between them, simultaneously segments demonstrated manipulations into sub-tasks. This study proposes a simple yet robust task decomposition method based on gaze transitions. Using teleoperation, a common modality in robotic manipulation for collecting demonstrations, in which a human operator's gaze is measured and used for task decomposition as a substitute for an imitating agent's gaze. Our approach ensures consistent task decomposition across all demonstrations for each task, which is desirable in contexts such as machine learning. We evaluated the method across demonstrations of various tasks, assessing the characteristics and consistency of the resulting sub-tasks. Furthermore, extensive testing across different hyperparameter settings confirmed its robustness, making it adaptable to diverse robotic systems. Our code is available at https://github.com/crumbyRobotics/GazeTaskDecomp.

Figures (4)

• Figure 1: Gaze itself possesses an inherent structure in object manipulation, which enables task decomposition by detecting transitions between gaze landmarks during demonstration. (a) Gaze transition of a human teleoperating a robot. (b) Overview of our proposed method.
• Figure 4: An example of the change scores $s_{\mathrm{pos}}$ (in red) and $s_{\mathrm{feat}}$ (in blue) in WrapCandy task. $s_{\mathrm{feat}}$ is multiplied by 1000 so that both scores share the same scale. The horizontal lines indicate the thresholds $\theta_{\mathrm{pos}} = 50$ and $\theta_{\mathrm{feat}} = 0.05$.
• Figure 5: Typical task decomposition results using the proposed method for the tasks of WrapCandy (a), MoveFlask (b), and OpenCap (c). The unshaded square areas represent the images cropped around the gaze positions.
• Figure 9: Success rate of decomposed demonstrations with varying hyperparameters for each task. (top) Without the refinement process. (bottom) With the refinement process.