Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models

TL;DR

To resolve the inherent tension between the VFM's semantic focus and the need for pixel-level fidelity, the VFM-VAE decoder is redesigned with Multi-Scale Latent Fusion and Progressive Resolution Reconstruction blocks, enabling high-quality reconstruction from spatially coarse VFM features.

Abstract

The performance of Latent Diffusion Models (LDMs) is critically dependent on the quality of their visual tokenizer. While recent works have explored incorporating Vision Foundation Models (VFMs) via distillation, we identify a fundamental flaw in this approach: it inevitably weakens the robustness of alignment with the original VFM, causing the aligned latents to deviate semantically under distribution shifts. In this paper, we bypass distillation by proposing a more direct approach: Vision Foundation Model Variational Autoencoder (VFM-VAE). To resolve the inherent tension between the VFM's semantic focus and the need for pixel-level fidelity, we redesign the VFM-VAE decoder with Multi-Scale Latent Fusion and Progressive Resolution Reconstruction blocks, enabling high-quality reconstruction from spatially coarse VFM features. Furthermore, we provide a comprehensive analysis of representation dynamics during diffusion training, introducing the proposed SE-CKNNA metric as a more precise tool for this diagnosis. This analysis allows us to develop a joint tokenizer-diffusion alignment strategy that dramatically accelerates convergence. Our innovations in tokenizer design and training strategy lead to superior performance and efficiency: our system reaches a gFID (w/o CFG) of 2.20 in merely 80 epochs (a 10x speedup over prior tokenizers). With continued training to 640 epochs, it further attains a gFID (w/o CFG) of 1.62, establishing direct VFM integration as a superior paradigm for LDMs.

Figures (12)

• Figure 1: Comparison of VFM-VAE and Previous Visual Tokenizers for LDM.(a) Distillation-based approach: VAE variants vavaerepae distill advanced representation from VFM. (b) Our VFM-VAE: directly leverage frozen VFM as a part of VAE. (c) Combing our visual tokenizer with LDMs variants leads faster converge and advanced performance.
• Figure 2: Brittleness of aligned representations under semantic-preserving transformations. Specifically, CKNNA platonic values for VA-VAE vavae and SD-VAE ldm are computed with DINOv2-Large dinov2, while those for VFM-VAE are computed with SigLIP2-Large siglip2. Under semantic-preserving transformations, VFM-VAE demonstrates notably stronger alignment with VFM features than VA-VAE.
• Figure 3: Overview of VFM-VAE architecture design. The model couples a frozen VFM encoder with a multi-scale decoder to preserve semantic alignment and enable high-fidelity reconstruction.
• Figure 4: CKNNA comparison across layers of the generative models. (a) Without explicit VFM alignment, the diffusion model combined with VFM-VAE achieves higher average layer-wise CKNNA than other tokenizer baselines. (b) With alignment enabled, the VFM-VAE system further improves, consistently surpassing other diffusion-aligned baselines in CKNNA.
• Figure 5: Stage-wise visualization of generative model training results. Shown under a fixed random seed and identical initial noise, our approach demonstrates impressive performance and greatly accelerates image generation learning.