Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Selective classification using a robust meta-learning approach

Nishant Jain, Karthikeyan Shanmugam, Pradeep Shenoy

TL;DR

Predictive uncertainty is essential for robust performance and selective classification. The authors propose ReVaR, a unified framework that learns an instance-conditioned uncertainty score (U-Score) via a bilevel optimization, jointly training a classifier and a meta-uncertainty model to minimize dropout-variance surrogate uncertainty. A variance-minimizing meta-regularizer drives the U-Score to capture diverse uncertainty sources, enabling effective test-time uncertainty measurement and training-time reweighting. Empirically, ReVaR achieves state-of-the-art selective classification and competitive calibration across diverse datasets and domain shifts, including large pretrained models like PLEX, highlighting its practical impact for robust AI systems.

Abstract

Predictive uncertainty-a model's self awareness regarding its accuracy on an input-is key for both building robust models via training interventions and for test-time applications such as selective classification. We propose a novel instance-conditioned reweighting approach that captures predictive uncertainty using an auxiliary network and unifies these train- and test-time applications. The auxiliary network is trained using a meta-objective in a bilevel optimization framework. A key contribution of our proposal is the meta-objective of minimizing the dropout variance, an approximation of Bayesian Predictive uncertainty. We show in controlled experiments that we effectively capture the diverse specific notions of uncertainty through this meta-objective, while previous approaches only capture certain aspects. These results translate to significant gains in real-world settings-selective classification, label noise, domain adaptation, calibration-and across datasets-Imagenet, Cifar100, diabetic retinopathy, Camelyon, WILDs, Imagenet-C,-A,-R, Clothing1M, etc. For Diabetic Retinopathy, we see upto 3.4%/3.3% accuracy and AUC gains over SOTA in selective classification. We also improve upon large-scale pretrained models such as PLEX.

Selective classification using a robust meta-learning approach

TL;DR

Predictive uncertainty is essential for robust performance and selective classification. The authors propose ReVaR, a unified framework that learns an instance-conditioned uncertainty score (U-Score) via a bilevel optimization, jointly training a classifier and a meta-uncertainty model to minimize dropout-variance surrogate uncertainty. A variance-minimizing meta-regularizer drives the U-Score to capture diverse uncertainty sources, enabling effective test-time uncertainty measurement and training-time reweighting. Empirically, ReVaR achieves state-of-the-art selective classification and competitive calibration across diverse datasets and domain shifts, including large pretrained models like PLEX, highlighting its practical impact for robust AI systems.

Abstract

Predictive uncertainty-a model's self awareness regarding its accuracy on an input-is key for both building robust models via training interventions and for test-time applications such as selective classification. We propose a novel instance-conditioned reweighting approach that captures predictive uncertainty using an auxiliary network and unifies these train- and test-time applications. The auxiliary network is trained using a meta-objective in a bilevel optimization framework. A key contribution of our proposal is the meta-objective of minimizing the dropout variance, an approximation of Bayesian Predictive uncertainty. We show in controlled experiments that we effectively capture the diverse specific notions of uncertainty through this meta-objective, while previous approaches only capture certain aspects. These results translate to significant gains in real-world settings-selective classification, label noise, domain adaptation, calibration-and across datasets-Imagenet, Cifar100, diabetic retinopathy, Camelyon, WILDs, Imagenet-C,-A,-R, Clothing1M, etc. For Diabetic Retinopathy, we see upto 3.4%/3.3% accuracy and AUC gains over SOTA in selective classification. We also improve upon large-scale pretrained models such as PLEX.
Paper Structure (42 sections, 17 equations, 3 figures, 27 tables, 1 algorithm)

This paper contains 42 sections, 17 equations, 3 figures, 27 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. ReVaR: dropout variance reduction
  4. Preliminaries & objective
  5. The learned reweighting framework
  6. Instance-conditional weights in ReVaR
  7. Variance minimization as meta-regularization
  8. Meta-learning with bilevel loss
  9. U-Score captures different sources of uncertainty
  10. Experiments and Results
  11. Baselines.
  12. Primary Real-world Setting: In-Domain Validation, In/Out-Domain Test
  13. ReVaR outperforms SOTA at selective classification
  14. ReVaR is competitive at calibration
  15. Input dependent label noise
  16. ...and 27 more sections

Figures (3)

  • Figure 1: Scenario 2 and 4 analysis with increasing distribution shift
  • Figure 2: Lesion Study (DR)
  • Figure 3: Selective classification results on diabetic retinopathy dataset. ReVaR shows robust improvement in AUC in both in-domain and domain-shift scenarios (panels (a,b)). Accuracy measures also show similar trends, with large improvements in domain shift conditions (panels (c,d)). Finally, selective calibration error measures (calibration of selected data points, panels (e,f)) show that better calibration is a key underlying factor for ReVaR's performance. See text for details.