Goal State Generation for Robotic Manipulation Based on Linguistically Guided Hybrid Gaussian Diffusion

TL;DR

This work tackles the challenge of generating precise, collision-free target states for robotic manipulation under language constraints. It advances a linguistically guided hybrid Gaussian diffusion (LHGD) framework conditioned on language, augmented by a gravity descent coverage (GDC) post-processing step to remove overlaps. Key contributions include decoupled diffusion for translation and rotation using a mixture noise model, a CLIP-backed language conditioning pipeline with Agent Attention for efficient fusion, and a GDC-based method to select collision-free targets, validated on a dataset of mugs, racks, and language cues. The proposed approach yields higher success rates across single- and multi-modal as well as language-specified tasks and directly provides collision-free targets suitable for downstream motion planning, reducing or eliminating the need for additional obstacle avoidance steps.

Abstract

In robotic manipulation tasks, achieving a designated target state for the manipulated object is often essential to facilitate motion planning for robotic arms. Specifically, in tasks such as hanging a mug, the mug must be positioned within a feasible region around the hook. Previous approaches have enabled the generation of multiple feasible target states for mugs; however, these target states are typically generated randomly, lacking control over the specific generation locations. This limitation makes such methods less effective in scenarios where constraints exist, such as hooks already occupied by other mugs or when specific operational objectives must be met. Moreover, due to the frequent physical interactions between the mug and the rack in real-world hanging scenarios, imprecisely generated target states from end-to-end models often result in overlapping point clouds. This overlap adversely impacts subsequent motion planning for the robotic arm. To address these challenges, we propose a Linguistically Guided Hybrid Gaussian Diffusion (LHGD) network for generating manipulation target states, combined with a gravity coverage coefficient-based method for target state refinement. To evaluate our approach under a language-specified distribution setting, we collected multiple feasible target states for 10 types of mugs across 5 different racks with 10 distinct hooks. Additionally, we prepared five unseen mug designs for validation purposes. Experimental results demonstrate that our method achieves the highest success rates across single-mode, multi-mode, and language-specified distribution manipulation tasks. Furthermore, it significantly reduces point cloud overlap, directly producing collision-free target states and eliminating the need for additional obstacle avoidance operations by the robotic arm.

Figures (9)

• Figure 1: Diagram of generation controlled by linguistic conditions (left) and illustration of de-overlapping module (right)
• Figure 2: Model Overview(reverse diffusion phase). During the training (diffusion) phase, we obtain the initial pose by adding noise to the target pose. An encoder is used to encode point cloud, pose, language, and timestep information into the feature space, which is then fused to obtain the complete feature information corresponding to the task in the scene. This information is then fed into the MLP structure combined with Agent Attention to predict the added noise. During the inference (reverse diffusion) phase, the trained parameters are used to predict the noise added to any initial mug pose, gradually restoring the target pose.
• Figure 3: The schematic diagram of the target state correction method based on the gravitational descent coverage coefficient. Here, $T_{0}^{\prime}$ represents the target pose obtained through network inference, and $T_{nc}^{\prime}$ represents the collision-free pose. We render the overlapped portion of the mug with the rack in red after the mug descends to $z_{\mathrm{opt}}$ under the collision-free pose, and render in blue the volume of this portion mapped onto the mug under $T_{nc}^{\prime}$. Finally, optimization is performed to obtain the best non-overlapping target state.
• Figure 4: The training mug models (top), the training rack models (middle), and the testing mug models (bottom).
• Figure 5: Visualization results of different methods on multi-modal distribution tasks.