Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation

TL;DR

TokenBridge resolves the long-standing trade-off between discrete and continuous image tokens by applying post-training, dimension-wise quantization to pretrained continuous VAE features, producing discrete tokens without retraining. It then models the resulting large token space with a lightweight dimension-wise autoregressive head that preserves inter-channel dependencies while remaining computationally feasible. The approach achieves reconstruction quality on par with continuous methods and competitive generation results on ImageNet-256, often surpassing Pure discrete baselines in efficiency and control capabilities. This work demonstrates that discrete-token autoregressive models can attain high fidelity without complex distribution modeling, suggesting a practical path toward simpler, scalable visual generation and multimodal integration.

Abstract

Autoregressive visual generation models typically rely on tokenizers to compress images into tokens that can be predicted sequentially. A fundamental dilemma exists in token representation: discrete tokens enable straightforward modeling with standard cross-entropy loss, but suffer from information loss and tokenizer training instability; continuous tokens better preserve visual details, but require complex distribution modeling, complicating the generation pipeline. In this paper, we propose TokenBridge, which bridges this gap by maintaining the strong representation capacity of continuous tokens while preserving the modeling simplicity of discrete tokens. To achieve this, we decouple discretization from the tokenizer training process through post-training quantization that directly obtains discrete tokens from continuous representations. Specifically, we introduce a dimension-wise quantization strategy that independently discretizes each feature dimension, paired with a lightweight autoregressive prediction mechanism that efficiently model the resulting large token space. Extensive experiments show that our approach achieves reconstruction and generation quality on par with continuous methods while using standard categorical prediction. This work demonstrates that bridging discrete and continuous paradigms can effectively harness the strengths of both approaches, providing a promising direction for high-quality visual generation with simple autoregressive modeling. Project page: https://yuqingwang1029.github.io/TokenBridge.

Figures (10)

• Figure 1: Comparison of different autoregressive visual generation approaches. (a) Traditional discrete tokenization incorporate quantization during training, resulting in tokenizer training instability and limited vocabulary size that restricts representational capacity. (b) Hybrid continuous AR models preserve rich visual information but need complex distribution modeling (diffusion or GMM) beyond standard categorical prediction. (c) Our approach bridges these paradigms by applying post-training quantization to pretrained continuous features, maintaining the high representational capacity of continuous tokens while enabling simple autoregressive modeling.
• Figure 2: Generated samples from TokenBridge. Class-conditional generation results on ImageNet imagenet 256×256 demonstrating fine details and textures across diverse categories including animals, food, objects, and scenes.
• Figure 3: Illustration of our post-training quantization process. The top row shows the pretrained continuous VAE tokenizer, mapping an input image to continuous latent features ${\bm{X}} \in \mathbb{R}^{H\times W\times C}$ and reconstructing it through the decoder. Our post-training quantization process (middle) transforms these continuous features into discrete tokens by independently quantizing each channel dimension. The bottom-left shows how our approach preserves the original Gaussian-like distribution (purple curve) in discretized form (purple histogram). The right portion demonstrates the de-quantization process that maps indices back to continuous values for decoding.
• Figure 4: Our autoregressive generation process. At the spatial level, our model autoregressively generates tokens conditioning on previous positions. For each spatial location (highlighted in pink), we apply dimension-wise sequential prediction to efficiently handle the large token space. This approach decomposes the modeling of each token into a series of smaller classification problems while preserving essential inter-dimensional dependencies.
• Figure 5: Reconstruction quality of typical continuous and discrete tokenizers. For discrete baselines, we use VQ from sun2024autoregressive, and LFQ from openmagvit2. Our method achieves reconstruction quality comparable to continuous VAE, preserving more fine details than traditional discrete tokenizers, especially in text and facial features. Zoom in for better comparison.