ArtiLatent: Realistic Articulated 3D Object Generation via Structured Latents

TL;DR

ArtiLatent introduces a unified diffusion-based framework for generating articulated 3D objects with fine-grained geometry, articulation semantics, and realistic appearance. By embedding voxel-level geometry, part semantics, and articulation attributes into a single latent space and applying a diffusion prior, the method learns physically plausible articulated structures, while an articulation-aware Gaussian decoder renders textures that adapt to articulation-induced visibility changes. A targeted articulation-aware fine-tuning strategy aligns appearance with multi-state renders, yielding improved texture fidelity on interior and exterior surfaces across articulation states. Evaluations on PartNet-Mobility and ACD demonstrate state-of-the-art geometric coherence and appearance realism, and the framework supports conditional generation from a single real image, enabling scalable articulated 3D content creation and interactive editing for embodied AI and digital twin applications.

Abstract

We propose ArtiLatent, a generative framework that synthesizes human-made 3D objects with fine-grained geometry, accurate articulation, and realistic appearance. Our approach jointly models part geometry and articulation dynamics by embedding sparse voxel representations and associated articulation properties, including joint type, axis, origin, range, and part category, into a unified latent space via a variational autoencoder. A latent diffusion model is then trained over this space to enable diverse yet physically plausible sampling. To reconstruct photorealistic 3D shapes, we introduce an articulation-aware Gaussian decoder that accounts for articulation-dependent visibility changes (e.g., revealing the interior of a drawer when opened). By conditioning appearance decoding on articulation state, our method assigns plausible texture features to regions that are typically occluded in static poses, significantly improving visual realism across articulation configurations. Extensive experiments on furniture-like objects from PartNet-Mobility and ACD datasets demonstrate that ArtiLatent outperforms existing approaches in geometric consistency and appearance fidelity. Our framework provides a scalable solution for articulated 3D object synthesis and manipulation.

Figures (6)

• Figure 1: Method overview. Given voxel-level articulation-aware inputs (occupancy, semantics, joint types, bounding boxes, joint parameters, and motion ranges), we encode them into a latent representation using an articulation-aware VAE. A conditional diffusion model samples articulation-aware latent codes under user-specified conditions (e.g., image), which are then decoded into an animatable voxel structure. The final appearance is generated using an articulation-aware Gaussian decoder, producing high-fidelity 3D Gaussian splats with consistent geometry and appearance across motion states.
• Figure 2: Effect of articulation-aware fine-tuning on appearance quality. We compare the results with (a, c) and without (b, d) articulation-aware fine-tuning on two different object types. Without fine-tuning, the generated textures exhibit noticeable artifacts, such as distortion, color bleeding, and loss of structure in articulated regions (see blue arrows). In contrast, our fine-tuned model produces sharper, more consistent, and plausible textures, especially around seams and occluded parts revealed by motion.
• Figure 3: Articulation-aware Gaussian decoder. We render the generated 3D Gaussians under multiple articulated states and use the corresponding images to supervise the encoder-decoder pair ($\mathcal{E}, \mathcal{D}_{\mathrm{GS}}$). For each voxel, we extract DINOv2 features across states and views, and use them as the initial feature. Decoded Gaussians are transformed according to articulation parameters, enabling the model to learn articulation-aware appearance variations via reconstruction loss.
• Figure 4: Qualitative comparison across different categories from the PartNet-Mobility and ACD datasets. The first column shows the input image, and the second column visualizes the ground-truth abstract articulation as a reference. Each object is displayed in both its resting and articulated states.
• Figure 5: Failure cases due to incorrect part segmentation. We show two examples where inaccurate voxel-part segmentation leads to unrealistic articulation. In both cases, the generated objects in the rest state (a, c) appear structurally correct, but in the articulated state (b, d), incorrect part grouping results in implausible deformations and motion artifacts (highlighted with blue arrows).