LoST: Level of Semantics Tokenization for 3D Shapes

Abstract

Tokenization is a fundamental technique in the generative modeling of various modalities. In particular, it plays a critical role in autoregressive (AR) models, which have recently emerged as a compelling option for 3D generation. However, optimal tokenization of 3D shapes remains an open question. State-of-the-art (SOTA) methods primarily rely on geometric level-of-detail (LoD) hierarchies, originally designed for rendering and compression. These spatial hierarchies are often token-inefficient and lack semantic coherence for AR modeling. We propose Level-of-Semantics Tokenization (LoST), which orders tokens by semantic salience, such that early prefixes decode into complete, plausible shapes that possess principal semantics, while subsequent tokens refine instance-specific geometric and semantic details. To train LoST, we introduce Relational Inter-Distance Alignment (RIDA), a novel 3D semantic alignment loss that aligns the relational structure of the 3D shape latent space with that of the semantic DINO feature space. Experiments show that LoST achieves SOTA reconstruction, surpassing previous LoD-based 3D shape tokenizers by large margins on both geometric and semantic reconstruction metrics. Moreover, LoST achieves efficient, high-quality AR 3D generation and enables downstream tasks like semantic retrieval, while using only 0.1%-10% of the tokens needed by prior AR models.

Figures (7)

• Figure 1: LoST, a novel shape tokenization that orders tokens by semantic salience, such that early prefixes decode into complete, plausible shapes that possess principal semantics, while subsequent tokens refine instance-specific geometric and semantic details. LoST produces prefix-decodable codes that boost semantic and geometric reconstruction over spatial level-of-detail baselines, while achieving much higher token efficiency using far fewer tokens.
• Figure 2: Overview of LoST. Left: LoST maps 3D shape latents into a token sequence ordered by semantic salience, where early prefixes capture coarse semantics and later tokens refine instance-specific detail. A conditional generative DiT decoder reconstructs the complete latent from any prefix. Right: The semantic extractor is pretrained with Relational Inter-Distance Alignment (RIDA), which aligns relationships in 3D latent space with DINO feature relationships to provide semantics-aware supervision.
• Figure 3: For each 3D shape (in blue), we visualize the shapes (in yellow) decoded from the learned LoST token sequences. Even as few as 1 token based generations result in semantically similar shapes while more tokens help to capture both semantic and geometric details.
• Figure 4: AR generation comparison. We compare image/text based autoregressive generation methods. LoST achieves superior performance in high-quality and faithful generation.
• Figure 5: Given a query shape (top), we show shape retrieval results using triplane, DINO, and RIDA features. While original triplane features focus on geometric similarity, RIDA mapped triplane features capture semantic alignment similar to DINO. In this example, we use a confusing query of a submarine shaped like a fish.