MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers

TL;DR

MeshGPT addresses the challenge of generating compact, artist-like triangle meshes directly rather than relying on post-processed neural fields. It learns a vocabulary of geometric embeddings from a graph-convolution encoder, quantizes them with residual vector quantization to produce a concise token sequence, and uses a GPT-style decoder to autoregressively generate meshes as sequences of tokens that are decoded into triangles. The approach yields sharper edges, improved shape coverage, and better perceptual quality on ShapeNetV2 relative to state-of-the-art baselines, while enabling novel shape generation and completion. Practically, MeshGPT offers a direct, controllable mesh generation paradigm with clear advantages in triangulation patterns and downstream rendering compatibility, albeit with slower sampling speeds and potential gains from scaling to larger models.

Abstract

We introduce MeshGPT, a new approach for generating triangle meshes that reflects the compactness typical of artist-created meshes, in contrast to dense triangle meshes extracted by iso-surfacing methods from neural fields. Inspired by recent advances in powerful large language models, we adopt a sequence-based approach to autoregressively generate triangle meshes as sequences of triangles. We first learn a vocabulary of latent quantized embeddings, using graph convolutions, which inform these embeddings of the local mesh geometry and topology. These embeddings are sequenced and decoded into triangles by a decoder, ensuring that they can effectively reconstruct the mesh. A transformer is then trained on this learned vocabulary to predict the index of the next embedding given previous embeddings. Once trained, our model can be autoregressively sampled to generate new triangle meshes, directly generating compact meshes with sharp edges, more closely imitating the efficient triangulation patterns of human-crafted meshes. MeshGPT demonstrates a notable improvement over state of the art mesh generation methods, with a 9% increase in shape coverage and a 30-point enhancement in FID scores across various categories.

Figures (16)

• Figure 1: Our method creates triangle meshes by autoregressively sampling from a transformer model that has been trained to produce tokens from a learned geometric vocabulary. These tokens can then be decoded into the faces of a triangle mesh. Our method generates clean, coherent, and compact meshes, characterized by sharp edges and high fidelity.
• Figure 2: Meshes generated by our method (top) for chairs, tables, benches, and lamps when trained on ShapeNet shapenet2015. MeshGPT meshes tend to be compact, with the ability to represent both sharp details and curved boundaries. This contrasts with neural field-based approaches that yield dense triangulations not easily simplified through decimation (bottom).
• Figure 3: We employ a graph convolutional encoder to process mesh faces, leveraging geometric neighborhood information to capture strong features representing intricate details of 3D shapes. These features are then quantized into codebook embeddings using residual quantization juang1982multiplemartinez2014stacked. In contrast to naive vector quantization, this ensures better reconstruction quality. The quantized embeddings are subsequently sequenced and decoded through a 1D ResNet he2016deep, guided by a reconstruction loss.
• Figure 4: Our method utilizes a ResNet he2016deep decoder that outputs mesh faces as a distribution over discretized coordinate values (center), as opposed to regression of continuous values (left). This significantly reduces floating face artifacts, leading to reconstructions that more closely resemble the ground truth (right).
• Figure 5: We employ a transformer to generate mesh sequences as token indices from a pre-learned codebook vocabulary. During training, a graph encoder extracts features from mesh faces, which are quantized into a set of face embeddings. These embeddings are flattened, bookended with start and end tokens, and fed into a GPT-style transformer. This decoder predicts the subsequent codebook index for each embedding, optimized via cross-entropy loss.