DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Data

TL;DR

This work tackles the challenge of controllable articulated 3D object generation from dual-state image pairs. It introduces DIPO, a diffusion-transformer generator conditioned on resting and articulated images, augmented by a Chain-of-Thought graph Reasoner to infer connectivity and guide layout generation. To address data scarcity and diversity, the authors develop LEGO-Art and the PM-X dataset, enabling scalable synthesis of complex articulated assets with URDF annotations and descriptions. Empirical results demonstrate state-of-the-art reconstruction quality, articulation accuracy, and generalization to out-of-distribution objects, validating the proposed data-augmented, dual-state conditioning approach.

Abstract

We present DIPO, a novel framework for the controllable generation of articulated 3D objects from a pair of images: one depicting the object in a resting state and the other in an articulated state. Compared to the single-image approach, our dual-image input imposes only a modest overhead for data collection, but at the same time provides important motion information, which is a reliable guide for predicting kinematic relationships between parts. Specifically, we propose a dual-image diffusion model that captures relationships between the image pair to generate part layouts and joint parameters. In addition, we introduce a Chain-of-Thought (CoT) based graph reasoner that explicitly infers part connectivity relationships. To further improve robustness and generalization on complex articulated objects, we develop a fully automated dataset expansion pipeline, name LEGO-Art, that enriches the diversity and complexity of PartNet-Mobility dataset. We propose PM-X, a large-scale dataset of complex articulated 3D objects, accompanied by rendered images, URDF annotations, and textual descriptions. Extensive experiments demonstrate that DIPO significantly outperforms existing baselines in both the resting state and the articulated state, while the proposed PM-X dataset further enhances generalization to diverse and structurally complex articulated objects. Our code and dataset will be released to the community upon publication.

Figures (12)

• Figure 1: Visual comparison on real-captured data. (a) SINGAPO struggles with challenging data and fails to model motion relationships due to its reliance on a single input. However, our DIPO (b), which conditioned on dual-state image pairs, effectively generates accurate layouts and enables precise control if part motion across different articulated states.
• Figure 2:
• Figure 3: Dual-state visual prompt used by the Graph Reasoner. GPT-4o can produce realistic and structurally complex image pairs.
• Figure 4: An overview of the fully automated synthesis pipeline for the proposed PM-X dataset. The synthesis pipeline consists of five functional modules executed in sequence: (1) a description roller that uses an LLM to generate natural language descriptions for structured layout, (2) a layout builder to generate part-level grid occupancy and joint configurations, (3) a scripting toolkit to construct precise coordinate from the grid-based layout information, (4) a retrieval and render module to assemble geometry and render dual-state images, and (5) a visual filter that uses a VLM to validate the plausibility of generated samples. In particular, modules (1), (2), and (5) are automatically constructed and managed by the AI Agent Designer.
• Figure 5: Visual comparison between the proposed DIPO and two baselines. The fist two columns show the dual-state image pairs. The precdiction results of articulate graph, the part layout and joint visualization in resting state, and the final geometry in articulated state are also illustrated. The first three rows are sampled from the PM dataset, the middle three rows are from the ACD dataset, and the last three rows are real-world images. Incorrect parts connections are marked with red box.