SFTok: Bridging the Performance Gap in Discrete Tokenizers

TL;DR

This work tackles the gap in discrete image tokenizers for high-resolution multimodal generation by introducing SFTok, a discrete tokenizer that uses a multi-step, consistency-aware training scheme. Central to SFTok are the self-forcing guided visual reconstruction (SFVR) strategy and a three-stage debias‑and‑fitting training protocol, designed to align training distributions with inference dynamics. Empirical results on ImageNet show state-of-the-art reconstruction at 64 tokens (rFID=1.21) and strong class-specific image generation (gFID=2.29), with ablations confirming the importance of SFVR, warm-up, and distribution alignment. The approach advances discrete-token-based multimodal modeling, offering competitive reconstruction and generation performance while maintaining the autoregressive advantages of discrete representations.

Abstract

Recent advances in multimodal models highlight the pivotal role of image tokenization in high-resolution image generation. By compressing images into compact latent representations, tokenizers enable generative models to operate in lower-dimensional spaces, thereby improving computational efficiency and reducing complexity. Discrete tokenizers naturally align with the autoregressive paradigm but still lag behind continuous ones, limiting their adoption in multimodal systems. To address this, we propose \textbf{SFTok}, a discrete tokenizer that incorporates a multi-step iterative mechanism for precise reconstruction. By integrating \textbf{self-forcing guided visual reconstruction} and \textbf{debias-and-fitting training strategy}, SFTok resolves the training-inference inconsistency in multi-step process, significantly enhancing image reconstruction quality. At a high compression rate of only 64 tokens per image, SFTok achieves state-of-the-art reconstruction quality on ImageNet (rFID = 1.21) and demonstrates exceptional performance in class-to-image generation tasks (gFID = 2.29).

Figures (17)

• Figure 1: SFTok employs self-forcing guided visual reconstruction (SFVR) that mitigates the training-inference inconsistency in multi-step iterative modeling within discrete tokenizers. (a) The image reconstruction quality progressively improves with the increase of inference steps. (b) High-quality image reconstruction is achieved at only 64 tokens (rFID = 1.21), outperforming other SOTA methods by a notable margin.
• Figure 2: Different mask token replacement strategies. (a) Vanilla strategy, where some masked tokens are replaced with the ground truth ${\bm{m}}_g$. (b) Our proposed SFVR training strategy, where some masked tokens are replaced with the model's first-step prediction $\hat{{\bm{m}}}_1$, ensuring training-inference consistency.
• Figure 3: Model architecture for SFTok. SFTok consists of an encoder, a quantizer, a decoder and a teacher model. Image features, along with query features, are processed by the ViT encoder and quantized into discrete tokens. The decoder then predicts the masked tokens with discrete ones through a multi-step iterative process, and the teacher model converts the predicted tokens into reconstructed images.
• Figure 4: (a) Visualization of KL divergence and Top-1 accuracy at each prediction step $\hat{{\bm{m}}}_i$, compared to the ground truth ${\bm{m}}_g$ and the final prediction $\hat{{\bm{m}}}_T$ during multi-step prediction. This illustrates the core cause of the training-inference inconsistency. (b) Variation curves of reconstruction loss with and without warming-up training, demonstrating the necessity of warming-up.
• Figure 5: Visualizations of reconstruction results on the ImageNet validation set, with detailed comparisons highlighted in red boxes.