Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ODP-Bench: Benchmarking Out-of-Distribution Performance Prediction

Han Yu, Kehan Li, Dongbai Li, Yue He, Xingxuan Zhang, Peng Cui

TL;DR

This work tackles the challenge of predicting a trained model's performance on unlabeled out-of-distribution (OOD) data. It introduces ODP-Bench, a large-scale benchmark comprising 29 OOD datasets, 1,444 off-the-shelf models, 10 prediction algorithms, and a codebase for extensibility, enabling fair and reproducible comparisons. Experimental results show strong predictive signals on synthetic corruptions but limited performance on complex real-world shifts, especially subpopulation shifts, underscoring gaps in current approaches. The benchmark provides a practical path toward reliable model deployment in risk-sensitive contexts by standardizing evaluation and facilitating systematic analysis of OOD performance prediction methods.

Abstract

Recently, there has been gradually more attention paid to Out-of-Distribution (OOD) performance prediction, whose goal is to predict the performance of trained models on unlabeled OOD test datasets, so that we could better leverage and deploy off-the-shelf trained models in risk-sensitive scenarios. Although progress has been made in this area, evaluation protocols in previous literature are inconsistent, and most works cover only a limited number of real-world OOD datasets and types of distribution shifts. To provide convenient and fair comparisons for various algorithms, we propose Out-of-Distribution Performance Prediction Benchmark (ODP-Bench), a comprehensive benchmark that includes most commonly used OOD datasets and existing practical performance prediction algorithms. We provide our trained models as a testbench for future researchers, thus guaranteeing the consistency of comparison and avoiding the burden of repeating the model training process. Furthermore, we also conduct in-depth experimental analyses to better understand their capability boundary.

ODP-Bench: Benchmarking Out-of-Distribution Performance Prediction

TL;DR

This work tackles the challenge of predicting a trained model's performance on unlabeled out-of-distribution (OOD) data. It introduces ODP-Bench, a large-scale benchmark comprising 29 OOD datasets, 1,444 off-the-shelf models, 10 prediction algorithms, and a codebase for extensibility, enabling fair and reproducible comparisons. Experimental results show strong predictive signals on synthetic corruptions but limited performance on complex real-world shifts, especially subpopulation shifts, underscoring gaps in current approaches. The benchmark provides a practical path toward reliable model deployment in risk-sensitive contexts by standardizing evaluation and facilitating systematic analysis of OOD performance prediction methods.

Abstract

Recently, there has been gradually more attention paid to Out-of-Distribution (OOD) performance prediction, whose goal is to predict the performance of trained models on unlabeled OOD test datasets, so that we could better leverage and deploy off-the-shelf trained models in risk-sensitive scenarios. Although progress has been made in this area, evaluation protocols in previous literature are inconsistent, and most works cover only a limited number of real-world OOD datasets and types of distribution shifts. To provide convenient and fair comparisons for various algorithms, we propose Out-of-Distribution Performance Prediction Benchmark (ODP-Bench), a comprehensive benchmark that includes most commonly used OOD datasets and existing practical performance prediction algorithms. We provide our trained models as a testbench for future researchers, thus guaranteeing the consistency of comparison and avoiding the burden of repeating the model training process. Furthermore, we also conduct in-depth experimental analyses to better understand their capability boundary.

Paper Structure

This paper contains 38 sections, 1 equation, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Out-of-Distribution Generalization
  4. Out-of-Distribution Performance Prediction
  5. Notations and Problem
  6. Notations
  7. Problem
  8. Benchmark Design
  9. Datasets
  10. Training
  11. Algorithms
  12. Evaluation Metrics
  13. Experiments
  14. Benchmark results
  15. Analysis of evaluation metrics
  16. ...and 23 more sections

Figures (5)

  • Figure 1: Rank correlation between predicted scores and ground truth performances. We can see that most of current algorithms do well in synthetic corruptions, but generally fail in natural shifts.
  • Figure 2: Scatter plots of predicted scores against ground truth accuracies.
  • Figure 3: Analyses of pretraining strategies, model architectures, and degree of distribution shift.
  • Figure 4: Distribution of real performances and predicted scores.
  • Figure 5: Calibration on ImageNet, ImageNet-R, and RxRx1.