Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Your VAR Model is Secretly an Efficient and Explainable Generative Classifier

Yi-Chung Chen, David I. Inouye, Jing Gao

TL;DR

This work shifts scalable generative classification from diffusion backbones to a visual autoregressive backbone by introducing Adaptive VAR Classifier (A-VARC) and its enhanced version A-VARC+. Through likelihood smoothing, partial-scale pruning, and Condition Contrastive Alignment finetuning, A-VARC+ achieves near diffusion-level accuracy on ImageNet-100 with up to 160x faster inference. The VAR framework exposes tractable likelihood, enabling token-wise visual explanations via token-wise PMI and offering inherent resistance to catastrophic forgetting in class-incremental learning. While not matching diffusion models on distribution-shift robustness, the approach provides a practical, explainable, and scalable alternative that broadens the study of generative classifiers and Their applications. Overall, VAR-based generative classifiers present a compelling path toward efficient, interpretable, and continually adaptable image classification.

Abstract

Generative classifiers, which leverage conditional generative models for classification, have recently demonstrated desirable properties such as robustness to distribution shifts. However, recent progress in this area has been largely driven by diffusion-based models, whose substantial computational cost severely limits scalability. This exclusive focus on diffusion-based methods has also constrained our understanding of generative classifiers. In this work, we propose a novel generative classifier built on recent advances in visual autoregressive (VAR) modeling, which offers a new perspective for studying generative classifiers. To further enhance its performance, we introduce the Adaptive VAR Classifier$^+$ (A-VARC$^+$), which achieves a superior trade-off between accuracy and inference speed, thereby significantly improving practical applicability. Moreover, we show that the VAR-based method exhibits fundamentally different properties from diffusion-based methods. In particular, due to its tractable likelihood, the VAR-based classifier enables visual explainability via token-wise mutual information and demonstrates inherent resistance to catastrophic forgetting in class-incremental learning tasks.

Your VAR Model is Secretly an Efficient and Explainable Generative Classifier

TL;DR

This work shifts scalable generative classification from diffusion backbones to a visual autoregressive backbone by introducing Adaptive VAR Classifier (A-VARC) and its enhanced version A-VARC+. Through likelihood smoothing, partial-scale pruning, and Condition Contrastive Alignment finetuning, A-VARC+ achieves near diffusion-level accuracy on ImageNet-100 with up to 160x faster inference. The VAR framework exposes tractable likelihood, enabling token-wise visual explanations via token-wise PMI and offering inherent resistance to catastrophic forgetting in class-incremental learning. While not matching diffusion models on distribution-shift robustness, the approach provides a practical, explainable, and scalable alternative that broadens the study of generative classifiers and Their applications. Overall, VAR-based generative classifiers present a compelling path toward efficient, interpretable, and continually adaptable image classification.

Abstract

Generative classifiers, which leverage conditional generative models for classification, have recently demonstrated desirable properties such as robustness to distribution shifts. However, recent progress in this area has been largely driven by diffusion-based models, whose substantial computational cost severely limits scalability. This exclusive focus on diffusion-based methods has also constrained our understanding of generative classifiers. In this work, we propose a novel generative classifier built on recent advances in visual autoregressive (VAR) modeling, which offers a new perspective for studying generative classifiers. To further enhance its performance, we introduce the Adaptive VAR Classifier (A-VARC), which achieves a superior trade-off between accuracy and inference speed, thereby significantly improving practical applicability. Moreover, we show that the VAR-based method exhibits fundamentally different properties from diffusion-based methods. In particular, due to its tractable likelihood, the VAR-based classifier enables visual explainability via token-wise mutual information and demonstrates inherent resistance to catastrophic forgetting in class-incremental learning tasks.
Paper Structure (23 sections, 12 equations, 7 figures, 7 tables, 1 algorithm)

This paper contains 23 sections, 12 equations, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative Classifier
  4. Image Autoregressive Model
  5. Preliminary
  6. Generative Classifier
  7. Diffusion Classifier
  8. Image Autoregressive Model
  9. Adaptive VAR Classifier
  10. Likelihood Smoothing
  11. Partial-Scale Candidate Pruning
  12. Class Information Enhancement via CCA
  13. Comparative Analysis
  14. Intriguing Properties of VAR-based Classifer
  15. Visual Explainability
  16. ...and 8 more sections

Figures (7)

  • Figure 1: Visual comparison of images with and without noise. Although the perturbation is not visually apparent, it alters 69% of the tokens, resulting in divergence of the estimated likelihood.
  • Figure 2: Relationship between top-10 accuracy and runtime as a function of the number of scales used in partial scale likelihood approximation.
  • Figure 3: Visual explanation of the VAR-based classifier using PMI. From left to right: the input image, PMI conditioned on the true label, PMI conditioned on the highest-ranked incorrect label, and the contrastive explanation between them.
  • Figure 4: Confusion matrices of the VAR classifier evaluated on the first 10 classes of ImageNet.
  • Figure 5: Impact of CCA finetuning. From left to right: the input image, PMI conditioned on the true label, PMI conditioned on the highest-ranked incorrect label, and the contrastive explanation between them.
  • ...and 2 more figures