ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding

TL;DR

ShapeLLM-Omni introduces a native multimodal LLM capable of text-to-3D, image-to-3D, 3D understanding, and 3D editing within a single autoregressive framework. It leverages a 3D VQVAE to map meshes to 1024 discrete tokens and trains on the large 3D-Alpaca dataset (3.46B tokens, 3D tasks across generation, understanding, and editing), finetuning a Qwen-2.5-VL-Instruct-7B backbone. The approach achieves competitive language performance while adding robust 3D generation/editing capabilities, with qualitative results showing structured geometry and editable assets. The work also provides a substantial 3D-focused training corpus and discusses limitations related to resource constraints and model scale, outlining directions toward more capable native 3D multimodal AI.

Abstract

Recently, the powerful text-to-image capabilities of ChatGPT-4o have led to growing appreciation for native multimodal large language models. However, its multimodal capabilities remain confined to images and text. Yet beyond images, the ability to understand and generate 3D content is equally crucial. To address this gap, we propose ShapeLLM-Omni-a native 3D large language model capable of understanding and generating 3D assets and text in any sequence. First, we train a 3D vector-quantized variational autoencoder (VQVAE), which maps 3D objects into a discrete latent space to achieve efficient and accurate shape representation and reconstruction. Building upon the 3D-aware discrete tokens, we innovatively construct a large-scale continuous training dataset named 3D-Alpaca, encompassing generation, comprehension, and editing, thus providing rich resources for future research and training. Finally, by performing instruction-based training of the Qwen-2.5-vl-7B-Instruct model on the 3D-Alpaca dataset. Our work provides an effective attempt at extending multimodal models with basic 3D capabilities, which contributes to future research in 3D-native AI. Project page: https://github.com/JAMESYJL/ShapeLLM-Omni

Figures (15)

• Figure 1: ShapeLLM-Omni inherits Qwen2.5-vl’s strong multimodal capabilities and additionally supports text-to-3D, image-to-3D, 3D captioning, and 3D editing using text instruction.
• Figure 2: The pipeline of 3D VQVAE, which can compress voxels into discrete tokens.
• Figure 3: Our proposed 3D-Alpaca dataset comprises 3D generation, 3D understanding, and 3D editing components, providing a comprehensive foundation for training and evaluating 3D large language models.
• Figure 4: Comparisons with other baselines on the image-to-3D task. Our results demonstrate more complete geometry and high-fidelity textures compared to baselines, enabling photorealistic image-to-3D generation.
• Figure 5: Comparisons with other baselines on text-to-3d task. Compared to other methods, our results show better text alignment, with generated 3D shapes accurately reflecting the input descriptions.