Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rejuvenating image-GPT as Strong Visual Representation Learners

Sucheng Ren, Zeyu Wang, Hongru Zhu, Junfei Xiao, Alan Yuille, Cihang Xie

TL;DR

D-iGPT revitalizes autoregressive pretraining for vision by replacing pixel targets with semantic tokens and adding supervision on visible tokens. The model uses a ViT-based encoder with two lightweight decoders and CLIP-derived tokens, optimizing $L_G$ and $L_D$ to learn rich visual representations. On ImageNet-1K, a ViT-B baseline reaches 86.2% top-1, with ViT-L achieving 87.8%, while pretraining on ImageNet-21K drives top-1 accuracy up to 90.0% for ViT-H, all using public data. The approach also demonstrates strong segmentation and zero-shot capabilities, as well as robustness to out-of-domain shifts, highlighting autoregressive pretraining as a scalable, effective paradigm for vision in the public-data regime.

Abstract

This paper enhances image-GPT (iGPT), one of the pioneering works that introduce autoregressive pretraining to predict the next pixels for visual representation learning. Two simple yet essential changes are made. First, we shift the prediction target from raw pixels to semantic tokens, enabling a higher-level understanding of visual content. Second, we supplement the autoregressive modeling by instructing the model to predict not only the next tokens but also the visible tokens. This pipeline is particularly effective when semantic tokens are encoded by discriminatively trained models, such as CLIP. We introduce this novel approach as D-iGPT. Extensive experiments showcase that D-iGPT excels as a strong learner of visual representations: A notable achievement is its compelling performance on the ImageNet-1K dataset -- by training on publicly available datasets, D-iGPT unprecedentedly achieves \textbf{90.0\%} top-1 accuracy with a vanilla ViT-H. Additionally, D-iGPT shows strong generalization on the downstream task. Code is available at https://github.com/OliverRensu/D-iGPT.

Rejuvenating image-GPT as Strong Visual Representation Learners

TL;DR

D-iGPT revitalizes autoregressive pretraining for vision by replacing pixel targets with semantic tokens and adding supervision on visible tokens. The model uses a ViT-based encoder with two lightweight decoders and CLIP-derived tokens, optimizing and to learn rich visual representations. On ImageNet-1K, a ViT-B baseline reaches 86.2% top-1, with ViT-L achieving 87.8%, while pretraining on ImageNet-21K drives top-1 accuracy up to 90.0% for ViT-H, all using public data. The approach also demonstrates strong segmentation and zero-shot capabilities, as well as robustness to out-of-domain shifts, highlighting autoregressive pretraining as a scalable, effective paradigm for vision in the public-data regime.

Abstract

This paper enhances image-GPT (iGPT), one of the pioneering works that introduce autoregressive pretraining to predict the next pixels for visual representation learning. Two simple yet essential changes are made. First, we shift the prediction target from raw pixels to semantic tokens, enabling a higher-level understanding of visual content. Second, we supplement the autoregressive modeling by instructing the model to predict not only the next tokens but also the visible tokens. This pipeline is particularly effective when semantic tokens are encoded by discriminatively trained models, such as CLIP. We introduce this novel approach as D-iGPT. Extensive experiments showcase that D-iGPT excels as a strong learner of visual representations: A notable achievement is its compelling performance on the ImageNet-1K dataset -- by training on publicly available datasets, D-iGPT unprecedentedly achieves \textbf{90.0\%} top-1 accuracy with a vanilla ViT-H. Additionally, D-iGPT shows strong generalization on the downstream task. Code is available at https://github.com/OliverRensu/D-iGPT.
Paper Structure (34 sections, 9 equations, 4 figures, 10 tables)

This paper contains 34 sections, 9 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Self-supervised Learning
  4. Discriminative Self-supervised Learning.
  5. Generatieve Self-supervised Learning.
  6. ImageNet-1K Winning Solutions
  7. Method
  8. Revisiting iGPT
  9. GPT.
  10. Image GPT.
  11. D-iGPT
  12. Modification I: semantic tokens.
  13. Mofication II: supervision on visible clusters.
  14. Summary.
  15. Architecture design.
  16. ...and 19 more sections

Figures (4)

  • Figure 1: ImageNet performance of models trained on publicly available datasets. We note that D-iGPT with ViT-H achieves the best performance, i.e., 90.0% top-1 accuracy.
  • Figure 2: The overview illustration of D-iGPT.
  • Figure 3: The model and dataset scalability of D-iGPT. D-iGPT shows significant performance gain with the growth of model size and dataset size.
  • Figure :