PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Model

TL;DR

PartRM introduces a 4D reconstruction framework that jointly models appearance, geometry, and part-level motion from multi-view observations by leveraging 3D Gaussian priors. It pairs a novel PartDrag-4D dataset with a multi-scale drag embedding and a two-stage training regime to learn motion first and appearance second, mitigating catastrophic forgetting. The approach achieves state-of-the-art performance on part-level motion learning, demonstrates temporal and multi-view consistency under drag, and enables sim-to-real manipulation in robotics. This work offers a practical path toward fast, 3D-consistent world models suitable for embodied AI tasks and provides public data, code, and models to spur further research.

Abstract

As interest grows in world models that predict future states from current observations and actions, accurately modeling part-level dynamics has become increasingly relevant for various applications. Existing approaches, such as Puppet-Master, rely on fine-tuning large-scale pre-trained video diffusion models, which are impractical for real-world use due to the limitations of 2D video representation and slow processing times. To overcome these challenges, we present PartRM, a novel 4D reconstruction framework that simultaneously models appearance, geometry, and part-level motion from multi-view images of a static object. PartRM builds upon large 3D Gaussian reconstruction models, leveraging their extensive knowledge of appearance and geometry in static objects. To address data scarcity in 4D, we introduce the PartDrag-4D dataset, providing multi-view observations of part-level dynamics across over 20,000 states. We enhance the model's understanding of interaction conditions with a multi-scale drag embedding module that captures dynamics at varying granularities. To prevent catastrophic forgetting during fine-tuning, we implement a two-stage training process that focuses sequentially on motion and appearance learning. Experimental results show that PartRM establishes a new state-of-the-art in part-level motion learning and can be applied in manipulation tasks in robotics. Our code, data, and models are publicly available to facilitate future research.

Figures (13)

• Figure 1: Introduction of PartDrag-4D dataset. PartDrag-4D utilizes 738 meshes spanning 8 categories to generate 20,548 articulation states. For each state, PartDrag-4D renders 12 views. The drags are sampled on the moving surface.
• Figure 2: Overview of PartRM. We first leverage a fine-tuned Zero123++ to generate multi-view images, followed by our designed drag propagation module to distribute drags on the moving parts. The drags and multi-view images are then fed into our designed network, where the drags are embedded using our multi-scale embedding module and subsequently concatenate to the UNet down blocks. We adopt a two-stage training approach: in the first stage, the network learns part motion using ground truth deformed 3D Gaussians as supervision, which are stored in the Gaussian database constructed by LGM. In the second stage, the network learns appearance, with ground truth deformed multi-view renderings serving as supervision.
• Figure 3: Illustration of drag propagation module.
• Figure 4: Illustration of drag embedding module.
• Figure 5: Qualitative comparisons between PartRM and baselines. The time values separated by the slash represent the time spent first applying 2D drag deformation to the input image and then performing 3D reconstruction using LGM and the optimization-based method, respectively. PartRM has only one value in the time column because it simultaneously models appearance, geometry, and part-level motion, eliminating the need for separate steps. PartRM learn the part motion effectively.