Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective

Yongxin Zhu, Bocheng Li, Hang Zhang, Xin Li, Linli Xu, Lidong Bing

TL;DR

Motivated by the underperformance of autoregressive image models despite sharing latent spaces with diffusion-based methods, the paper reframes latent-space design around stability. It introduces DiGIT, a discriminative self-supervised tokenizer that discretizes a stable latent space learned by a DINOv2-based encoder, enabling GPT-style next-token image generation with strong results in both understanding and generation. Empirical evidence shows DiGIT achieves state-of-the-art linear-probe performance and, at scale, surpasses latent-diffusion baselines in image generation, with substantial FID improvements and high quality IS. The work highlights the pivotal role of latent-space stability and discrete tokenization for scalable, GPT-like vision models, suggesting a new direction for autoregressive pre-training in images.

Abstract

Latent-based image generative models, such as Latent Diffusion Models (LDMs) and Mask Image Models (MIMs), have achieved notable success in image generation tasks. These models typically leverage reconstructive autoencoders like VQGAN or VAE to encode pixels into a more compact latent space and learn the data distribution in the latent space instead of directly from pixels. However, this practice raises a pertinent question: Is it truly the optimal choice? In response, we begin with an intriguing observation: despite sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in image generation. This finding contrasts sharply with the field of NLP, where the autoregressive model GPT has established a commanding presence. To address this discrepancy, we introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling. Furthermore, we propose a simple but effective discrete image tokenizer to stabilize the latent space for image generative modeling by applying K-Means on the latent features of self-supervised learning models. Experimental results show that image autoregressive modeling with our tokenizer (DiGIT) benefits both image understanding and image generation with the next token prediction principle, which is inherently straightforward for GPT models but challenging for other generative models. Remarkably, for the first time, a GPT-style autoregressive model for images outperforms LDMs, which also exhibits substantial improvement akin to GPT when scaling up model size. Our findings underscore the potential of an optimized latent space and the integration of discrete tokenization in advancing the capabilities of image generative models. The code is available at \url{https://github.com/DAMO-NLP-SG/DiGIT}.

Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective

TL;DR

Motivated by the underperformance of autoregressive image models despite sharing latent spaces with diffusion-based methods, the paper reframes latent-space design around stability. It introduces DiGIT, a discriminative self-supervised tokenizer that discretizes a stable latent space learned by a DINOv2-based encoder, enabling GPT-style next-token image generation with strong results in both understanding and generation. Empirical evidence shows DiGIT achieves state-of-the-art linear-probe performance and, at scale, surpasses latent-diffusion baselines in image generation, with substantial FID improvements and high quality IS. The work highlights the pivotal role of latent-space stability and discrete tokenization for scalable, GPT-like vision models, suggesting a new direction for autoregressive pre-training in images.

Abstract

Latent-based image generative models, such as Latent Diffusion Models (LDMs) and Mask Image Models (MIMs), have achieved notable success in image generation tasks. These models typically leverage reconstructive autoencoders like VQGAN or VAE to encode pixels into a more compact latent space and learn the data distribution in the latent space instead of directly from pixels. However, this practice raises a pertinent question: Is it truly the optimal choice? In response, we begin with an intriguing observation: despite sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in image generation. This finding contrasts sharply with the field of NLP, where the autoregressive model GPT has established a commanding presence. To address this discrepancy, we introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling. Furthermore, we propose a simple but effective discrete image tokenizer to stabilize the latent space for image generative modeling by applying K-Means on the latent features of self-supervised learning models. Experimental results show that image autoregressive modeling with our tokenizer (DiGIT) benefits both image understanding and image generation with the next token prediction principle, which is inherently straightforward for GPT models but challenging for other generative models. Remarkably, for the first time, a GPT-style autoregressive model for images outperforms LDMs, which also exhibits substantial improvement akin to GPT when scaling up model size. Our findings underscore the potential of an optimized latent space and the integration of discrete tokenization in advancing the capabilities of image generative models. The code is available at \url{https://github.com/DAMO-NLP-SG/DiGIT}.

Paper Structure

This paper contains 26 sections, 4 theorems, 16 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Latent Space for Generative Models
  4. Distance between distributions in different spaces.
  5. Optimal Latent Space for Generative Models.
  6. Two Pathways of the Latent Space Construction.
  7. Stability of the Latent space
  8. Stabilize the Latent Space with Self-supervised Learning Model
  9. Discrete Image Discriminative Tokenizer
  10. Image Autoregressive Modeling
  11. Experiments
  12. Implementation Details
  13. Image Understanding
  14. Image Generation
  15. Ablation Study
  16. ...and 11 more sections

Key Result

Proposition 2.1

The latent space spanned by a linear autoencoder is congruent with that spanned by the principal component loading vectors derived in Principal Component Analysis (PCA). Furthermore, the principal component loading vectors can be elucidated from the autoencoder's weights.

Figures (8)

  • Figure 1: (a): Linear probe and class-unconditional generation performance of different methods trained and evaluated on ImageNet-1K. (b): Class-conditional generation performance of different methods on ImageNet-1k. The size of the bubbles indicates the number of parameters in the models. DiGIT achieves SOTA performance in linear probing and establishes a new SOTA in image generation within a single model.
  • Figure 2: The architecture of DiGIT.
  • Figure 3: Ablation study of DiGIT. (a) The comparison of tokenizer, training steps, and model size in the image generation task. (b) Linear-probe accuracy from different layers in the pre-trained DiGIT-base with different number of K-Means clusters.
  • Figure 4: (a): The comparison of tokenizers induced from different SSL models. Acc@LP is obtained by linear probing on the autoregressive model (model size of 39M for 100 epochs) trained with tokenizers. Acc@OL is the linear probe score of the SSL model. "P": patch, "D": discriminative, "R": reconstructive. (b): Generation quality curves in FID on ImageNet $256\times 256$ valid set when scaling the prefix length with discriminative tokenizer and reconstructive VQGAN tokenizer. Both are autoregressive models with 219M parameters.
  • Figure 5: FID and Inception Score as a function of top-k, top-p sampling on the image generation task with DiGIT-base. The decoding temperature is fixed to 1.0. The "stage2" denotes the autoregressive model for pixel rendering.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Proposition 2.1
  • Proposition 2.2
  • Proposition A.1
  • proof
  • Proposition A.2
  • proof