Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration

Hiroki Naganuma, Ryuichiro Hataya, Kotaro Yoshida, Ioannis Mitliagkas

TL;DR

This paper systematically evaluates how pre-trained model choice—across size, pre-training data, and training strategies—affects out-of-distribution generalization and confidence calibration in vision. By screening 100 pre-trained models on five distribution-shift benchmarks, it finds that model selection can yield larger gains than algorithmic improvements, with scaling trends showing that bigger models and larger pre-training datasets enhance both OOD accuracy and calibration. Vision Transformer and ConvNeXt architectures often outperform traditional CNNs under these settings, and calibration improves in tandem with OOD performance, challenging some IID-centric calibration assumptions. The authors present practical guidelines to prioritize pre-trained model selection (favor large models, large pre-training datasets, and generic pre-training methods) and advocate for treating model selection as a strong baseline in OOD research.

Abstract

In the field of computer vision, fine-tuning pre-trained models has become a prevalent strategy for out-of-distribution (OOD) generalization tasks. Different from most prior work that has focused on advancing learning algorithms, we systematically examined how pre-trained model size, pre-training dataset size, and training strategies impact generalization and confidence calibration on downstream tasks. We evaluated 100 models across diverse pre-trained model sizes, five pre-training datasets, and five data augmentations through extensive experiments on four distribution shift datasets totaling over 120,000 GPU hours. Our results demonstrate the significant impact of pre-trained model selection, with optimal choices substantially improving OOD accuracy over algorithm improvement alone. Additionally, we find that larger models and bigger pre-training datasets not only enhance OOD performance but also improve calibration, helping to mitigate overconfidence, contrary to some prior studies that found modern deep networks to calibrate worse than classical shallow models. Our work underscores the overlooked importance of pre-trained model selection for out-of-distribution generalization and calibration.

An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration

TL;DR

This paper systematically evaluates how pre-trained model choice—across size, pre-training data, and training strategies—affects out-of-distribution generalization and confidence calibration in vision. By screening 100 pre-trained models on five distribution-shift benchmarks, it finds that model selection can yield larger gains than algorithmic improvements, with scaling trends showing that bigger models and larger pre-training datasets enhance both OOD accuracy and calibration. Vision Transformer and ConvNeXt architectures often outperform traditional CNNs under these settings, and calibration improves in tandem with OOD performance, challenging some IID-centric calibration assumptions. The authors present practical guidelines to prioritize pre-trained model selection (favor large models, large pre-training datasets, and generic pre-training methods) and advocate for treating model selection as a strong baseline in OOD research.

Abstract

In the field of computer vision, fine-tuning pre-trained models has become a prevalent strategy for out-of-distribution (OOD) generalization tasks. Different from most prior work that has focused on advancing learning algorithms, we systematically examined how pre-trained model size, pre-training dataset size, and training strategies impact generalization and confidence calibration on downstream tasks. We evaluated 100 models across diverse pre-trained model sizes, five pre-training datasets, and five data augmentations through extensive experiments on four distribution shift datasets totaling over 120,000 GPU hours. Our results demonstrate the significant impact of pre-trained model selection, with optimal choices substantially improving OOD accuracy over algorithm improvement alone. Additionally, we find that larger models and bigger pre-training datasets not only enhance OOD performance but also improve calibration, helping to mitigate overconfidence, contrary to some prior studies that found modern deep networks to calibrate worse than classical shallow models. Our work underscores the overlooked importance of pre-trained model selection for out-of-distribution generalization and calibration.
Paper Structure (31 sections, 1 theorem, 4 equations, 28 figures, 4 tables)

This paper contains 31 sections, 1 theorem, 4 equations, 28 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Related Works
  4. Experimental Setup
  5. Results
  6. Scaling Trend: OOD Generalization
  7. A Deeper Analysis from the Perspective of Confidence Calibration
  8. Pre-trained Models can Overfit to Training Set
  9. Other Findings
  10. Discussion and Conclusion
  11. Detailed Experimental Settings
  12. Models used in the Experiments
  13. Model Selection and Hyperparameter Search
  14. Experimental Protocol of CLIP Model
  15. Datasets
  16. ...and 16 more sections

Key Result

Theorem E.1

Let $p(y | x)$ denote the model's predicted probability distribution over the target variable $y$ given input $x$, and let $P(y | x)$ represent the true conditional probability distribution. Suppose the following conditions hold: Then, under these assumptions, the model's predictions $p(y | x)$ converge to the true distribution $P(y | x)$, resulting in an ECE that approaches zero.

Figures (28)

  • Figure 1: This paper investigates the out-of-distribution (OOD) generalization and confidence calibration of a total of 100 ImageNet pre-trained models on five datasets from DomainBed Gulrajani21 and WILDS koh2021wilds. These panels show the relationship between expected calibration error (ECE) rates (lower is better) and OOD test accuracy (higher is better). Marker sizes are proportional to the number of model parameters, and marker shapes correspond to pre-training configurations. We can observe a general trend that larger models achieve the best of both worlds (bottom-right corner), except for VGGs and MLP-Mixers. The full legend can be found in \ref{['ap:fig:all-all-test-acc-test-ece']}.
  • Figure 2: The relationship between in-domain (ID) validation accuracy and out-of-distribution (OOD) test accuracy of all experiments. When ID validation accuracy is high enough, ID and OOD accuracy are highly correlated, the "training-domain validation set" scheme can select near-optimal models.
  • Figure 3: (a,b) OOD test error rates (lower is better) with respect to the number of parameters (a, log-scale) and the pre-training dataset sizes (b) on the PACS dataset. In the right panel, the same model architectures with different pre-training datasets are connected by dashed lines. We can observe that the number of parameters and the pre-training dataset sizes contribute to the OOD generalization. Presented models are selected for a better view. Results of other OOD datasets with all models are presented in \ref{['ap:fig:all-test-acc-test-ece', 'ap:fig:all-all-test-acc-test-ece', 'ap:fig:dataset-avg_test_acc-avg_test_ece']}. (c,d) OOD expected calibration error (lower is better) with respect to the number of parameters (c) and the pre-training dataset sizes (d) on the PACS dataset. In the right panel, the same model architectures with different pre-training datasets are connected by dashed lines. We can see trends that the number of parameters and the pre-training dataset sizes improve ECE. Presented models are selected for the better view. Results of other OOD datasets with all models are presented in \ref{['ap:fig:all-test-acc-test-ece', 'ap:fig:all-all-test-acc-test-ece', 'ap:fig:dataset-avg_test_acc-avg_test_ece']}.
  • Figure 4: Correlation between ImageNet-1k validation accuracy and out-of-distribution test accuracy. OOD test accuracy shows a strong positive correlation with ImageNet-1k validation accuracy. However, EfficientNets fall below the trend of other models, suggesting that they may be overfitting to ImageNet-1k.
  • Figure 5: Out-of-distribution test accuracy of ResNetshe2016deep with different training schemes. The results of the original ResNets are connected by dashed lines. Although extensive training recipes wightman2021resnet improve ImageNet-1k performance, their fine-tuned OOD results often underperform the original ones.
  • ...and 23 more figures

Theorems & Definitions (2)

  • Theorem E.1: informal
  • proof