Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation

Abstract

Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose \textbf{SemTok}, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.

Figures (12)

• Figure 1: An overview of SemTok. (a) SemTok compresses 2D images into 1D discrete tokens through the encoder and reconstructs images through the decoder. A semantic alignment constraint is introduced at the encoder end for compact semantic representations. (b) To explore the semantic diversity of the latent space, SemTok adopts a two-stage training strategy: in Stage I, a diffusion-based decoder is used to predict images from noise; in Stage II, a one-step refiner is trained on image reconstruction tasks to enhance texture details.
• Figure 2: Illustration of masked autoregressive modeling of image generation with SemTok. (left) During training, we randomly mask several tokens and utilize bidirectional attention to allow each token to see all tokens, thereby perceiving global semantics. (right) During inference, the model predicts multiple tokens simultaneously in a random order until the complete sequence is filled.
• Figure 3: Comparison of reconstructions from different tokenizers.
• Figure 4: Generation results with different tokenizers.
• Figure 5: Progressive generation results in raster scan order between 1D and 2D tokens.