CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models

TL;DR

CG-MLLM addresses the lack of end-to-end 3D reasoning in multimodal LLMs by introducing a Mixture-of-Transformers architecture that binds token-level and block-level generation to distinct objectives. By integrating a vision-language backbone (Qwen3-VL) with a high-order 3D latent space (Hunyuan3D-2.1-VAE) and employing modalities adapters for text, image, and 3D data, the model achieves high-fidelity, spatially consistent 3D generation within a single framework. It demonstrates superior performance on 3D understanding, captioning, and generation relative to prior MLLMs, with comprehensive quantitative metrics and qualitative assessments. Limitations include AdaLN-related stability issues, VAE reconstruction artifacts, and remaining gaps to specialized 3D generators, guiding future work toward scalable, unified multimodal generation.

Abstract

Large Language Models(LLMs) have revolutionized text generation and multimodal perception, but their capabilities in 3D content generation remain underexplored. Existing methods compromise by producing either low-resolution meshes or coarse structural proxies, failing to capture fine-grained geometry natively. In this paper, we propose CG-MLLM, a novel Multi-modal Large Language Model (MLLM) capable of 3D captioning and high-resolution 3D generation in a single framework. Leveraging the Mixture-of-Transformer architecture, CG-MLLM decouples disparate modeling needs, where the Token-level Autoregressive (TokenAR) Transformer handles token-level content, and the Block-level Autoregressive (BlockAR) Transformer handles block-level content. By integrating a pre-trained vision-language backbone with a specialized 3D VAE latent space, CG-MLLM facilitates long-context interactions between standard tokens and spatial blocks within a single integrated architecture. Experimental results show that CG-MLLM significantly outperforms existing MLLMs in generating high-fidelity 3D objects, effectively bringing high-resolution 3D content creation into the mainstream LLM paradigm.

Figures (8)

• Figure 1: The Pipeline of CG-MLLM. Our multimodal architecture processes vision, text, and 3D spatial inputs to generate text and 3D spatial outputs. It features a TokenAR Transformer for sequential next-token prediction and a BlockAR Transformer for efficient parallel block prediction, both governed by strict causal masking.
• Figure 2: Our approach unifies spatial perception and generation in a single model, supporting image understanding, point cloud understanding, mesh generation, and textual intent understanding across multiple spatial modalities.
• Figure 3: Example mask used in CG-MLLM.
• Figure 4: Comparison with other MLLM-based methods on the image-to-3D task. For clearer visualization of geometry, materials are removed from the second row. Our method generates more complete geometry, achieving photorealistic results.
• Figure 5: More Image-to-3D results produced by our method. For clearer visualization of geometry, materials are removed from the rightmost column. Zooming in is recommended for better inspection.