Language-Guided Object-Centric Diffusion Policy for Generalizable and Collision-Aware Robotic Manipulation

TL;DR

This paper presents Lan-o3dp, a language-guided, object-centric diffusion policy for robust and generalizable robotic manipulation. By conditioning a diffusion model on segmented 3D point clouds of target objects and incorporating a time-independent, cost-guided diffusion mechanism for obstacle avoidance, the approach achieves high data efficiency and strong zero-shot collision avoidance. The method leverages open-vocabulary segmentation and large language models to ground user instructions to objects and obstacles, enabling open-set manipulation and safety in cluttered and view-shifted scenes. Empirical results in RLBench show a substantial performance boost over 2D and full-scene 3D baselines, with successful real-world generalization across unseen configurations, demonstrating practical impact for flexible, safe robotic manipulation.

Abstract

Learning from demonstrations faces challenges in generalizing beyond the training data and often lacks collision awareness. This paper introduces Lan-o3dp, a language-guided object-centric diffusion policy framework that can adapt to unseen situations such as cluttered scenes, shifting camera views, and ambiguous similar objects while offering training-free collision avoidance and achieving a high success rate with few demonstrations. We train a diffusion model conditioned on 3D point clouds of task-relevant objects to predict the robot's end-effector trajectories, enabling it to complete the tasks. During inference, we incorporate cost optimization into denoising steps to guide the generated trajectory to be collision-free. We leverage open-set segmentation to obtain the 3D point clouds of related objects. We use a large language model to identify the target objects and possible obstacles by interpreting the user's natural language instructions. To effectively guide the conditional diffusion model using a time-independent cost function, we proposed a novel guided generation mechanism based on the estimated clean trajectories. In the simulation, we showed that diffusion policy based on the object-centric 3D representation achieves a much higher success rate (68.7%) compared to baselines with simple 2D (39.3%) and 3D scene (43.6%) representations across 21 challenging RLBench tasks with only 40 demonstrations. In real-world experiments, we extensively evaluated the generalization in various unseen situations and validated the effectiveness of the proposed zero-shot cost-guided collision avoidance.

Figures (8)

• Figure 1: An illustration of the proposed pipeline of Lan-o3dp. We use open-set segmentation to obtain the point clouds of objects. At the training stage, the visual observations in the demonstrations we collected only contained point clouds of objects relevant to the task. During deployment, a large language model is employed to decompose users' instructions into target objects and obstacles and select the corresponding policy given a set of trained policies. Target objects are used as visual observation for the model, while obstacles are transformed into a cost function to guide the model in generating collision-free trajectories.
• Figure 2: Visualization of some simulation tasks. We use a single front camera to keep consistent with the real world. The top and middle row visualize the observations of two baselines: 2D RGB images and 3D point clouds of the scene from the front camera. The bottom row shows visualizations of our method of object-centric 3D point clouds.
• Figure 3: Simulation results. (a) The average success rates over all 21 RLBench tasks. (b) The distribution of success rates. Our method with object-centric 3D representation achieves a higher average success rate and has a larger number of tasks in the 60-100% success rate range.
• Figure 4: Four tasks in real-world experiments: pouring, bottle upright, brushing, and tape to the drawer. The red lines indicate the initial position variations of the objects in the collected 40 demonstrations.
• Figure 5: Quantitative results of evaluating generalization. Our method has high success rates and strong generalization capability. Diffusion policy chi2023diffusionpolicy achieves bad results because of limited demonstrations and poor generalization capabilities.