Improving Flexible Image Tokenizers for Autoregressive Image Generation

TL;DR

This paper tackles the generation bottleneck in flexible 1D image tokenizers used for autoregressive image generation. It identifies that naive nested dropout concentrates information in early tokens, limiting gains when increasing token counts. The authors propose ReTok, featuring redundant token padding to activate tail tokens and hierarchical semantic regularization that aligns early-token decodings with vision foundation models while gradually relaxing constraints for tail tokens, plus decoder fine-tuning. On ImageNet 256×256, ReTok achieves state-of-the-art performance among flexible tokenizers and is competitive with fixed-length tokenizers, e.g., $gFID$ as low as $2.27$ with AR-XL; ablations confirm the importance of padding, semantic regularization, and decoder tuning. This work enhances practical AR generation with flexible tokenization and suggests avenues for scaling to higher resolutions and video tokens.

Abstract

Flexible image tokenizers aim to represent an image using an ordered 1D variable-length token sequence. This flexible tokenization is typically achieved through nested dropout, where a portion of trailing tokens is randomly truncated during training, and the image is reconstructed using the remaining preceding sequence. However, this tail-truncation strategy inherently concentrates the image information in the early tokens, limiting the effectiveness of downstream AutoRegressive (AR) image generation as the token length increases. To overcome these limitations, we propose \textbf{ReToK}, a flexible tokenizer with \underline{Re}dundant \underline{Tok}en Padding and Hierarchical Semantic Regularization, designed to fully exploit all tokens for enhanced latent modeling. Specifically, we introduce \textbf{Redundant Token Padding} to activate tail tokens more frequently, thereby alleviating information over-concentration in the early tokens. In addition, we apply \textbf{Hierarchical Semantic Regularization} to align the decoding features of earlier tokens with those from a pre-trained vision foundation model, while progressively reducing the regularization strength toward the tail to allow finer low-level detail reconstruction. Extensive experiments demonstrate the effectiveness of ReTok: on ImageNet 256$\times$256, our method achieves superior generation performance compared with both flexible and fixed-length tokenizers. Code will be available at: \href{https://github.com/zfu006/ReTok}{https://github.com/zfu006/ReTok}

Figures (12)

• Figure 1: Overview of our method. (a) Previous methods with naive nested dropout compress information into early tokens, compromising generation quality. (b) Our method with redundant token padding activates tail tokens, consistently improving generation quality as the token sequence extends. (c) Illustration of progressive generation process of our method with increasing tokens.
• Figure 2: Illustration of the generation bottleneck in flexible tokenizers. We train AR models and evaluate their generation quality at varying token lengths. The results indicate a significant decrease in downstream image generation compared to fixed-length tokenizers. Implementation details are introduced in \ref{['sec_finding']}.
• Figure 3: Ablation study on the improvement of ReTok. We evaluate the effectiveness of our proposed methods on the ImageNet validation set. By applying redundant token padding, hierarchical semantic regularization, and decoder fine-tuning, the generation quality improves under all token lengths compared to using naive nested dropout. \ref{['sec_exp_roadmap']} introduces the detailed implementation.
• Figure 4: Analysis of image generation under different token lengths. (a) Downstream image generation comparison of ReTok, DetailFlow liu2025detailflow, and One-D-Piece miwa2025one. (b) Generation performance of ReTok with various AR models.
• Figure 5: Image reconstruction comparison with various tokens. Low-resolution results of DetailFlow are resized to 256$\times$256. Our tokenizer show high-fidelity reconstruction at 32 and 256 tokens.