Leveraging Pretrained Diffusion Models for Zero-Shot Part Assembly

TL;DR

This work tackles zero-shot 3D part assembly by leveraging pre-trained point-cloud diffusion models as density priors to steer per-part pose transformations, without task-specific supervision. It formulates an iterative loop where diffusion-based denoising is combined with rigid ICP refinements and an explicit push-away collision strategy, grounded by a Score Distillation Sampling objective. The approach demonstrates strong zero-shot performance on PartNet chair assemblies, surpasses several baselines and even competes with some supervised methods, and generalizes to airplanes while also exploring 2D diffusion extensions. By reducing or eliminating the need for labeled pose data, this method offers a practical pathway toward scalable, robust assembly in real-world robotics and opens avenues for cross-domain diffusion-guided reassembly tasks.

Abstract

3D part assembly aims to understand part relationships and predict their 6-DoF poses to construct realistic 3D shapes, addressing the growing demand for autonomous assembly, which is crucial for robots. Existing methods mainly estimate the transformation of each part by training neural networks under supervision, which requires a substantial quantity of manually labeled data. However, the high cost of data collection and the immense variability of real-world shapes and parts make traditional methods impractical for large-scale applications. In this paper, we propose first a zero-shot part assembly method that utilizes pre-trained point cloud diffusion models as discriminators in the assembly process, guiding the manipulation of parts to form realistic shapes. Specifically, we theoretically demonstrate that utilizing a diffusion model for zero-shot part assembly can be transformed into an Iterative Closest Point (ICP) process. Then, we propose a novel pushing-away strategy to address the overlap parts, thereby further enhancing the robustness of the method. To verify our work, we conduct extensive experiments and quantitative comparisons to several strong baseline methods, demonstrating the effectiveness of the proposed approach, which even surpasses the supervised learning method. The code has been released on https://github.com/Ruiyuan-Zhang/Zero-Shot-Assembly.

Figures (12)

• Figure 1: The overall architecture of our algorithm. Given the misaligned input clouds $\mathcal{P}_t$, we introduce noise to the shape, which helps the diffusion model recognize the data. The diffusion process then refines the input, generating a point cloud closer to the target chair shape. To achieve rigid transformation, we apply the ICP method for alignment, producing updated pose vectors. By iterating this process over $T$ steps, the algorithm effectively assembles the disordered parts into the final coherent structure.
• Figure 2: Visual comparisons demonstrating our superior assembly performance over baseline methods on PartNet. The first column shows our input at the Excessive level, while the last column presents reference samples obtained through diffusion sampling.
• Figure 3: Different $z$ in our experiments.
• Figure 4: Different Views from Baseline-Simple and Ours. Baseline-Simple utilizes supervised learning on point clouds generated by a diffusion model, while our method employs density estimates. The results of the Simple are similar to set of point clouds from reference, but do not correspond to a chair shape.
• Figure 5: Ablation Study of Push Action. Based on our method, we can clearly separate overlapping parts, which helps reduce the overlap problem.