PnP-U3D: Plug-and-Play 3D Framework Bridging Autoregression and Diffusion for Unified Understanding and Generation

TL;DR

PnP-U3D addresses unified 3D understanding and generation by marrying autoregressive understanding with diffusion-based generation. It introduces a lightweight cross-modal transformer that bridges a frozen vision-language model and a pretrained 3D diffusion backbone, preserving priors while enabling efficient information exchange. The framework trains only small auxiliary components while leveraging strong pretrained priors, and it extends naturally to 3D editing via instruction-driven prompts. Experiments show state-of-the-art results across 3D understanding, text-to-3D generation, and 3D editing, highlighting AR+diffusion as a practical path toward general-purpose 3D intelligence.

Abstract

The rapid progress of large multimodal models has inspired efforts toward unified frameworks that couple understanding and generation. While such paradigms have shown remarkable success in 2D, extending them to 3D remains largely underexplored. Existing attempts to unify 3D tasks under a single autoregressive (AR) paradigm lead to significant performance degradation due to forced signal quantization and prohibitive training cost. Our key insight is that the essential challenge lies not in enforcing a unified autoregressive paradigm, but in enabling effective information interaction between generation and understanding while minimally compromising their inherent capabilities and leveraging pretrained models to reduce training cost. Guided by this perspective, we present the first unified framework for 3D understanding and generation that combines autoregression with diffusion. Specifically, we adopt an autoregressive next-token prediction paradigm for 3D understanding, and a continuous diffusion paradigm for 3D generation. A lightweight transformer bridges the feature space of large language models and the conditional space of 3D diffusion models, enabling effective cross-modal information exchange while preserving the priors learned by standalone models. Extensive experiments demonstrate that our framework achieves state-of-the-art performance across diverse 3D understanding and generation benchmarks, while also excelling in 3D editing tasks. These results highlight the potential of unified AR+diffusion models as a promising direction for building more general-purpose 3D intelligence.

Figures (12)

• Figure 1: PnP-U3D: the first unified 3D AR+Diffusion framework, enabling 3D understanding, generation, and editing.
• Figure 2: Qualitative text-to-3D generation results. PnP-U3D is capable of generating diverse and high-quality 3D models.
• Figure 3: Framework overview. PnP-U3D uses an autoregressive next-token paradigm for understanding and a continuous diffusion paradigm for generation. The two components interact via a trainable cross-modal connector.
• Figure 4: Illustration of our Text-3D and 3D Editing Datasets. The Text–3D Dataset pairing 3D shapes with multi-granular textual descriptions for 3D understanding and generation, and the 3D Editing Dataset pairing edited 3D shapes with editing prompts for instruction-driven editing.
• Figure 5: Qualitative comparison on 3D object captioning. Benefiting from the flexibility of our framework, combining additional 2D information with the 3D latent yields more accurate and detailed captions. In contrast, PointLLM xu2024pointllm often generates overly long outputs with meaningless redundancy and still suffers from hallucination issues. Note that our method may produce hallucinations due to the lack of texture information in the 3D latent.