LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models

TL;DR

The paper tackles enabling large language models to generate and understand 3D meshes directly from text. It introduces Llama-Mesh, which represents meshes as plain-text OBJ data quantized to 64 bins to avoid tokenizer changes. A supervised fine-tuning dataset of text-3D dialogues is used to fine-tune LLaMA-3.1-8B-Instruct, enabling text-to-mesh generation, interleaved outputs, and mesh understanding while preserving language capabilities. Results show mesh quality on par with models trained from scratch and demonstrate the feasibility of unifying language and 3D modalities in a single pretrained Transformer.

Abstract

This work explores expanding the capabilities of large language models (LLMs) pretrained on text to generate 3D meshes within a unified model. This offers key advantages of (1) leveraging spatial knowledge already embedded in LLMs, derived from textual sources like 3D tutorials, and (2) enabling conversational 3D generation and mesh understanding. A primary challenge is effectively tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly. To address this, we introduce LLaMA-Mesh, a novel approach that represents the vertex coordinates and face definitions of 3D meshes as plain text, allowing direct integration with LLMs without expanding the vocabulary. We construct a supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate 3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs as required, and (3) understand and interpret 3D meshes. Our work is the first to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge for 3D mesh generation in a text-based format, effectively unifying the 3D and text modalities. LLaMA-Mesh achieves mesh generation quality on par with models trained from scratch while maintaining strong text generation performance.

Figures (11)

• Figure 1: An illustration of our method, Llama-Mesh, which enables the generation of 3D meshes from human instructions via a conversational interface. Users provide textual prompts, and the model responds with both text and 3D mesh outputs, facilitating interactive 3D content creation. Llama-Mesh allows large language models to generate and interpret 3D meshes from text directly, seamlessly unifying language and 3D modalities within a single model.
• Figure 2: Overview of our method. Llama-Mesh unifies text and 3D mesh in a uniform format by representing the numerical values of vertex coordinates and face definitions of a 3D mesh as plain text. Our model is trained using text and 3D interleaved data end-to-end. Therefore, with a single, unified model, we can generate both text and 3D meshes.
• Figure 3: Gallery of generations from Llama-Mesh. We can generate high-quality and diverse meshes with artist-like created topology.
• Figure 4: Illustration of our 3D representation approach. Left: A snippet of an OBJ file represented as plain text, containing vertex (v) and face (f) definitions. Right: The 3D object rendered from the OBJ file. We enable the LLM to process and generate 3D meshes by converting the mesh data into a textual format.
• Figure 5: Illustration of our vertex quantization method.Top: The original OBJ file represents vertex coordinates in decimal values, splitting a single coordinate into several tokens. Bottom: After quantization, we represent the vertices as integers containing fewer tokens and are processed by LLM more efficiently.