Balancing Accuracy, Calibration, and Efficiency in Active Learning with Vision Transformers Under Label Noise

TL;DR

This study investigates how Vision Transformer architectures, across patch sizes and embedding capacities, perform in active learning under symmetric label noise on CIFAR-10/100. By comparing ViT variants (ViTb16, ViTb32, ViTl16, ViTl32) and SwinV2 models under Random, Entropy, and GCI_ViTAL acquisition strategies, the authors reveal that the largest ViT (ViTl32) typically achieves the best accuracy and calibration under moderate to high label noise, while smaller patch sizes do not consistently improve results and can incur higher costs. Information-based active learning strategies boost accuracy at moderate noise but often worsen calibration, with random sampling sometimes yielding superior calibration under heavy noise. The work provides practical guidance for deploying vision transformers in resource-constrained settings, highlighting that ViTl32 frequently offers the most favorable balance of performance and efficiency, and suggesting that calibration-focused considerations may favor simpler sampling strategies in high-noise regimes.

Abstract

Fine-tuning pre-trained convolutional neural networks on ImageNet for downstream tasks is well-established. Still, the impact of model size on the performance of vision transformers in similar scenarios, particularly under label noise, remains largely unexplored. Given the utility and versatility of transformer architectures, this study investigates their practicality under low-budget constraints and noisy labels. We explore how classification accuracy and calibration are affected by symmetric label noise in active learning settings, evaluating four vision transformer configurations (Base and Large with 16x16 and 32x32 patch sizes) and three Swin Transformer configurations (Tiny, Small, and Base) on CIFAR10 and CIFAR100 datasets, under varying label noise rates. Our findings show that larger ViT models (ViTl32 in particular) consistently outperform their smaller counterparts in both accuracy and calibration, even under moderate to high label noise, while Swin Transformers exhibit weaker robustness across all noise levels. We find that smaller patch sizes do not always lead to better performance, as ViTl16 performs consistently worse than ViTl32 while incurring a higher computational cost. We also find that information-based Active Learning strategies only provide meaningful accuracy improvements at moderate label noise rates, but they result in poorer calibration compared to models trained on randomly acquired labels, especially at high label noise rates. We hope these insights provide actionable guidance for practitioners looking to deploy vision transformers in resource-constrained environments, where balancing model complexity, label noise, and compute efficiency is critical in model fine-tuning or distillation.

Figures (15)

• Figure 1: The key components involved in fine-tuning a transformer under label noise. The aspects we vary in our experiments are indicated in red. The Active Learning variation spans the entire diagram. [Adapted from Mots'oehli:GCIViTAL24]
• Figure 2: This figure shows the second stage of the GCI_ViTAL query strategy, with C-Core attention vectors from the ViT model guiding the selection of semantically challenging samples based on their distance from class centroids. Label smoothing mitigates noise, enhancing model noise robustness Mots'oehli:GCIViTAL24. Steps 1-5 of the strategy are implemented as described in the original paper
• Figure 3: Top-1 Accuracy vs. Label Noise Rate averaged over CIFAR10 and CIFAR100 at 13% labeled data. Each subplot shows a different active learning strategy (random, entropy, GCI_ViTAL) as explained in Section \ref{['subsec:Active_Learning_Meth']}, across various Vision Transformer (ViT) and Swin Transformer (SwinV2) models. See Appendix \ref{['sec:appendix_a_results']} for the version of this graph split by dataset and at different labeled data proportions.
• Figure 4: The Brier Score vs. Noise Rate averaged over both CIFAR10 and CIFAR100 at 13% labeled data proportion. Each subplot compares multiple ViT and SwinV2 models under different Active learning strategies. We see an overall dominance of the ViT architecture over the SwinV2 variants. We also see marginally higher calibration using the random query over the information-based acquisition strategies. See appendix \ref{['sec:appendix_a_results']} for similar additional results
• Figure 5: Top-1 Accuracy vs. Noise Rate on CIFAR10 with 23% labeled data. Each subplot represents a different DAL strategy (random, entropy, GCI_ViTAL) applied to Vision Transformer (ViT) and Swin Transformer (Swin) models. The same trends seen at 13% labeled data persist across VIT size and AL strategy.