PhysPart: Physically Plausible Part Completion for Interactable Objects

TL;DR

PhysPart tackles physically plausible part completion for interactable objects using a diffusion-based generator that combines geometry-conditioned prompts with physics-aware losses during sampling. The method features a pose-proposal stage to estimate the missing part's bounding box and a latent SDF diffusion model trained with a 3D-VQVAE, augmented by loss-guided sampling to enforce stability and plausible motion. A physical-plausibility benchmark and motion-based metric demonstrate improvements over baselines, with practical demonstrations in 3D printing, robot manipulation, and sequential part generation. The work enables more realistic, manipulable object generation and lays groundwork for complex part-whole hierarchies in 3D modeling and robotics.

Abstract

Interactable objects are ubiquitous in our daily lives. Recent advances in 3D generative models make it possible to automate the modeling of these objects, benefiting a range of applications from 3D printing to the creation of robot simulation environments. However, while significant progress has been made in modeling 3D shapes and appearances, modeling object physics, particularly for interactable objects, remains challenging due to the physical constraints imposed by inter-part motions. In this paper, we tackle the problem of physically plausible part completion for interactable objects, aiming to generate 3D parts that not only fit precisely into the object but also allow smooth part motions. To this end, we propose a diffusion-based part generation model that utilizes geometric conditioning through classifier-free guidance and formulates physical constraints as a set of stability and mobility losses to guide the sampling process. Additionally, we demonstrate the generation of dependent parts, paving the way toward sequential part generation for objects with complex part-whole hierarchies. Experimentally, we introduce a new metric for measuring physical plausibility based on motion success rates. Our model outperforms existing baselines over shape and physical metrics, especially those that do not adequately model physical constraints. We also demonstrate our applications in 3D printing, robot manipulation, and sequential part generation, showing our strength in realistic tasks with the demand for high physical plausibility.

Figures (6)

• Figure 1: Pipeline of our proposed framework. We train a pose proposal model to predict the missing part's bounding box and a latent diffusion model conditioned on the input object's point cloud and the missing part's bounding box within the latent space of part SDF. During inference, the trained pose proposal model first predicts the missing part's bounding box. We then apply the proposed physical-aware losses (contact and collision losses in static or dynamics states) to guide the sampling process.
• Figure 2: Normalized 3D coordinate and shared motion constraints for certain part categories. Specifically, drawers could be pulled out along the $+z$ axis, while doors could rotate around $+x$, $-x$, $+y$, and $-y$ axes.
• Figure 3: Results of self-moving part generation. All visualizations are done in the objects' canonical poses, with the base shapes in green and the generated parts in red. While all methods can generate part shapes that are roughly reasonable, the bumpy surfaces and size mismatches in baseline results hinder their physical plausibility.
• Figure 4: Results of dependent part generation. All visualizations are done in the objects' canonical poses, with the base shapes in green and the generated parts in red.
• Figure 5: Downstream applications of our method: (a) 3D printing for real-world object completion, (b) sequential part generation for complex structures, and (c) simulation for robot manipulation.