Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding vision transformer robustness through the lens of out-of-distribution detection

Joey Kuang, Alexander Wong

TL;DR

This paper investigates how quantization to low precision affects vision transformers' robustness to distribution shifts. By evaluating small-variant ViTs (ViT, DeiT, DeiT3) quantized to $4$-bit with RepQ-ViT and tested on ImageNet-1k (ID) plus six OOD datasets, it reveals that large-scale pretraining on ImageNet-22k can degrade OOD detection performance, while data augmentation can help mitigate these effects. Attention-map analyses show that high-norm outlier tokens may dominate under low precision, offering a partial explanation for the observed OOD trends. The findings suggest that pretraining strategy and augmentation are crucial for maintaining OOD robustness in quantized ViTs, informing deployment choices beyond ID accuracy.

Abstract

Vision transformers have shown remarkable performance in vision tasks, but enabling them for accessible and real-time use is still challenging. Quantization reduces memory and inference costs at the risk of performance loss. Strides have been made to mitigate low precision issues mainly by understanding in-distribution (ID) task behaviour, but the attention mechanism may provide insight on quantization attributes by exploring out-of-distribution (OOD) situations. We investigate the behaviour of quantized small-variant popular vision transformers (DeiT, DeiT3, and ViT) on common OOD datasets. ID analyses show the initial instabilities of 4-bit models, particularly of those trained on the larger ImageNet-22k, as the strongest FP32 model, DeiT3, sharply drop 17% from quantization error to be one of the weakest 4-bit models. While ViT shows reasonable quantization robustness for ID calibration, OOD detection reveals more: ViT and DeiT3 pretrained on ImageNet-22k respectively experienced a 15.0% and 19.2% average quantization delta in AUPR-out between full precision to 4-bit while their ImageNet-1k-only counterparts experienced a 9.5% and 12.0% delta. Overall, our results suggest pretraining on large scale datasets may hinder low-bit quantization robustness in OOD detection and that data augmentation may be a more beneficial option.

Understanding vision transformer robustness through the lens of out-of-distribution detection

TL;DR

This paper investigates how quantization to low precision affects vision transformers' robustness to distribution shifts. By evaluating small-variant ViTs (ViT, DeiT, DeiT3) quantized to -bit with RepQ-ViT and tested on ImageNet-1k (ID) plus six OOD datasets, it reveals that large-scale pretraining on ImageNet-22k can degrade OOD detection performance, while data augmentation can help mitigate these effects. Attention-map analyses show that high-norm outlier tokens may dominate under low precision, offering a partial explanation for the observed OOD trends. The findings suggest that pretraining strategy and augmentation are crucial for maintaining OOD robustness in quantized ViTs, informing deployment choices beyond ID accuracy.

Abstract

Vision transformers have shown remarkable performance in vision tasks, but enabling them for accessible and real-time use is still challenging. Quantization reduces memory and inference costs at the risk of performance loss. Strides have been made to mitigate low precision issues mainly by understanding in-distribution (ID) task behaviour, but the attention mechanism may provide insight on quantization attributes by exploring out-of-distribution (OOD) situations. We investigate the behaviour of quantized small-variant popular vision transformers (DeiT, DeiT3, and ViT) on common OOD datasets. ID analyses show the initial instabilities of 4-bit models, particularly of those trained on the larger ImageNet-22k, as the strongest FP32 model, DeiT3, sharply drop 17% from quantization error to be one of the weakest 4-bit models. While ViT shows reasonable quantization robustness for ID calibration, OOD detection reveals more: ViT and DeiT3 pretrained on ImageNet-22k respectively experienced a 15.0% and 19.2% average quantization delta in AUPR-out between full precision to 4-bit while their ImageNet-1k-only counterparts experienced a 9.5% and 12.0% delta. Overall, our results suggest pretraining on large scale datasets may hinder low-bit quantization robustness in OOD detection and that data augmentation may be a more beneficial option.
Paper Structure (9 sections, 2 figures, 2 tables)

This paper contains 9 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Motivation
  4. Setup
  5. Key findings
  6. Vision transformers have unpredictable levels of robustness to 4-bit quantization
  7. Vision transformers pretrained on ImageNet-22k are less robust to quantization in OOD
  8. High-norm outlier tokens in attention maps may be too dominant in lower precision
  9. Conclusion

Figures (2)

  • Figure 1: Higher performing models assign OOD examples with higher anomaly scores. See Sec. \ref{['sec:methodology']} for a description of the AUPR-out as an OOD detection measure. Models pretrained on the larger dataset are more sensitive to quantization error in OOD detection.
  • Figure 2: DeiT touvron_training_2021 produces interpretable attention maps even at lower precision. We observe outlier artifacts similar to findings in darcet_vision_2023 in DeiT3 features. Despite poor AUPR robustness to 4-bit quantization, attention maps produced by DeiT3-S pretrained on IN22k seem stable. Attention maps do not appear to vary greatly under OOD data. Predicted class label and probability are labeled in each image.