You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

TL;DR

This work proposes a closed-loop, one-shot category-level manipulation framework that generalizes to novel object instances by learning a simulation-trained, object-centric representation (NUNOCS) and using a model-free 6 DoF motion tracker to extract a demonstration trajectory in $SE(3)$. The demonstrated trajectory is reprojected to unseen objects via a dynamic, category-level canonical frame and executed with CatBC, augmented by a local attention mechanism that selects task-relevant anchors throughout the manipulation horizon. The approach achieves robust, high-precision manipulation across long-horizon tasks in real-world high-precision assembly, significantly improving generalization and reducing data collection compared to prior category-level methods. The combination of simulation-trained NUNOCS, real-time visual feedback, and dynamic frame anchoring enables flexible, strategy-aware manipulation from a single example, with practical implications for rapid deployment in industrial settings.

Abstract

Promising results have been achieved recently in category-level manipulation that generalizes across object instances. Nevertheless, it often requires expensive real-world data collection and manual specification of semantic keypoints for each object category and task. Additionally, coarse keypoint predictions and ignoring intermediate action sequences hinder adoption in complex manipulation tasks beyond pick-and-place. This work proposes a novel, category-level manipulation framework that leverages an object-centric, category-level representation and model-free 6 DoF motion tracking. The canonical object representation is learned solely in simulation and then used to parse a category-level, task trajectory from a single demonstration video. The demonstration is reprojected to a target trajectory tailored to a novel object via the canonical representation. During execution, the manipulation horizon is decomposed into longrange, collision-free motion and last-inch manipulation. For the latter part, a category-level behavior cloning (CatBC) method leverages motion tracking to perform closed-loop control. CatBC follows the target trajectory, projected from the demonstration and anchored to a dynamically selected category-level coordinate frame. The frame is automatically selected along the manipulation horizon by a local attention mechanism. This framework allows to teach different manipulation strategies by solely providing a single demonstration, without complicated manual programming. Extensive experiments demonstrate its efficacy in a range of challenging industrial tasks in highprecision assembly, which involve learning complex, long-horizon policies. The process exhibits robustness against uncertainty due to dynamics as well as generalization across object instances and scene configurations. The supplementary video is available at https://www.youtube.com/watch?v=WAr8ZY3mYyw

Figures (13)

• Figure 1: During offline NUNOCS learning, the NUNOCS Net is trained using synthetic data generated using the training CAD models $\mathbb{O}_\text{train}$. The purpose of NUNOCS Net is to map an input point cloud to the Non-Uniform Normalized Object Coordinate Space (NUNOCS) for the object category, from which a 9D pose (translation, rotation and 3D scaling) of the observed instance in the category canonical frame can be solved in closed-form umeyama1991least. Upon demonstration, a model-free 6 DoF motion tracker parses the video and tracks the trajectory of the demonstrated object $\mathcal{O_D}$. This tracked trajectory is then lifted to a category-level demonstration trajectory by using the NUNOCS representation. In particular, the NUNOCS Net predicts the category-level object pose of the demonstrated object $\mathcal{O_D}$ in the first video frame. Given the 3D model of $\mathcal{O_D}$ and the category-level pose, a mapping $f_{\text{demo}}$ between the NUNOCS shape and the scanned cloud of $\mathcal{O_D}$ in the first video frame is obtained. During testing on a novel object $O$, the NUNOCS Net takes the scanned cloud and predicts the mapping $f_\text{test}$ to its category-level NUNOCS representation. It then establishes a dense correspondence between the NUNOCS representation of $O$ and the NUNOCS shape of $\mathcal{O_D}$ by finding nearest neighbors, which enables to transfer the demonstration category-level trajectory to a new trajectory tailored for the target novel object $O$. The manipulation process is split into long-range, collision-free motion and last-inch manipulation. For the latter part, category-level behavior cloning is employed, which aims to clone the target category-level trajectory. Visual feedback for this process is provided by a 6 DoF motion tracker and allows behavioral cloning to adapt the manipulation of the object so that it closely follows the target trajectory until task completion. Red arrows and text denote data flow that occurs exclusively offline.
• Figure 2: Left: Heatmap visualizations of the local attention mechanism are shown in the top and bottom rows for the training and testing objects respectively. During demonstration, given the 3D model of the demonstration object and its paired receptacle, an attention heatmap is precomputed. During online execution, the attention heatmap can be transferred to a novel object given the dense correspondence established through their NUNOCS representations shown in the middle row. The attention mechanism allows to dynamically anchor the coordinate system to the local attended point (located at the warmest color), capturing the variation between demonstration and testing objects in scaling and local typology. The testing objects' 3D models are shown for visualization only and are unknown during execution. Right: Experimental hardware setup.
• Figure 3: Experimental objects in categories Gears and Batteries. In each category, the testing object set are labeled with IDs. The rest are the training object instances with known CAD models. Note that the real world training objects are only used for data collection to train baseline methods. The proposed approach learns solely in simulation using their CAD models. The testing objects are selected to be manipulable with the gripper but otherwise diverse in shape and appearance cross instances.
• Figure 4: (a) Distribution of gears' initial poses relative to the receptacle in the "gear insertion" experiments (Sec. \ref{['sec:gear_task']}). The gray gear mesh represents the demonstration object $\mathcal{O_D}$ in its initial configuration. During testing, the framework generalizes to unseen configurations. (b) Overall success rates of the 3 policies learned from different demonstrations in the "gear insertion" task. The success rates are averaged across object instances and the same number of runs for the $0.5mm$ tolerance as in Table \ref{['tab:gear']}. The method Ours in Table \ref{['tab:gear']} is based on "human naive". (c) Running time of last-inch manipulation in different tasks. (d) An example testing case of the "battery assembly" task, where the proposed approach generalizes to unstructured environments with obstacles unseen in the demonstration video. (e) Visual motion tracker constantly updates the object pose for robust CatBC against external disturbances, such as human dragging. Pose visualization thumbnails are in bottom-left corners. (f) For the "gear insertion" task, an anchor-and-pivot manipulation strategy is provided instead (first row), and the robot executes the learned policy on a testing object (second row). Complete videos are available in supplementary material.
• Figure 5: Example failure modes. Top: In the "battery assembly" task, as the battery squeezes the spring, the elastic force gradually increases and eventually pushes the battery out of the gripper. In this case, it would be beneficial to add tactile sensing together with the visual tracker in the feedback loop to predict slippery early. Bottom: In the "gear insertion" task, when using “anchor-and-pivot” strategy, rich contact leads to significant change of the gear's in hand orientation, causing no collision-free IK solutions found to continue. In this scenario, the object is still accurately tracked, so regrasping or adjusting the object pose via in-hand manipulation with a dexterous hand can recover from this failure mode.