NativeTok: Native Visual Tokenization for Improved Image Generation

TL;DR

NativeTok tackles the misalignment between image tokenization and generation in VQ-based image synthesis by introducing native visual tokenization that enforces causal dependencies and an ordered token sequence. The framework combines a Meta Image Transformer (MIT) for latent image modeling with a Mixture of Causal Expert Transformer (MoCET) to generate tokens sequentially, guided by a Hierarchical Native Training strategy for scalable learning. Experiments on ImageNet-1K demonstrate substantial gains in generation quality under autoregressive and MaskGIT-style pipelines, with notable reductions in gFID and solid reconstruction metrics. By tightly coupling tokenization with the generation process, NativeTok enables more coherent and controllable image synthesis and informs future two-stage visual generative systems.

Abstract

VQ-based image generation typically follows a two-stage pipeline: a tokenizer encodes images into discrete tokens, and a generative model learns their dependencies for reconstruction. However, improved tokenization in the first stage does not necessarily enhance the second-stage generation, as existing methods fail to constrain token dependencies. This mismatch forces the generative model to learn from unordered distributions, leading to bias and weak coherence. To address this, we propose native visual tokenization, which enforces causal dependencies during tokenization. Building on this idea, we introduce NativeTok, a framework that achieves efficient reconstruction while embedding relational constraints within token sequences. NativeTok consists of: (1) a Meta Image Transformer (MIT) for latent image modeling, and (2) a Mixture of Causal Expert Transformer (MoCET), where each lightweight expert block generates a single token conditioned on prior tokens and latent features. We further design a Hierarchical Native Training strategy that updates only new expert blocks, ensuring training efficiency. Extensive experiments demonstrate the effectiveness of NativeTok.

Figures (8)

• Figure 1: Illustration of our motivation. (a) Existing disordered tokenization overlooks the essential requirement of relational modeling in the generation stage, as it fails to introduce any token dependency constraints during tokenization, resulting in a gap between the two stages of image generation. (b) Our approach proposes native visual tokenization, which considers not only reconstruction quality but also imposes relational constraints during tokenization, thereby coupling the two stages of image generation.
• Figure 2: The overview of our NativeTok framework. (a) In Meta Image Transformer (MIT), the image information is initially modeled by a Pixel Transformer, then compresses the image into the latent space. During the subsequent generation process, the latent space information of the image remains locked. (b) During the token sequence generation process, we define Mixture of Causal Expert Transformer (MoCET). The $i^{th}$ expert transformer block is responsible for generating the $i^{th}$ token. In each generation step, we concatenate the locked latent space image information, all previously generated tokens, and the current mask token, and feed them into the corresponding expert transformer. Once a token is generated, it remains fixed. (c) Once all tokens are generated, they are fed into the decoder to reconstruct the image.
• Figure 3: Hierarchical Native Training: We freeze the Meta Image Transformer and existing experts, training only newly added experts initialized with reused weights. This reduces training costs while ensuring the new experts inherit prior modeling capabilities.
• Figure 4: In (a), the top and bottom rows correspond to the original images and their reconstructed results. In (b), we showcase examples of generated images.
• Figure 5: Visualization: In the right-hand figure, the x-axis denotes the token position index, and the y-axis represents the corresponding probability values. When modifying adjacent token representations, we compare how the probability distribution of the next token changes accordingly.