Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making Reliable and Flexible Decisions in Long-tailed Classification

Bolian Li, Ruqi Zhang

TL;DR

The paper addresses the challenge of making reliable decisions in long-tailed classification by embedding decision risk into the learning objective via an integrated gain grounded in Bayesian Decision Theory. It introduces a flexible utility-based framework RF-DLC that aligns training with the testing distribution through importance weighting and learns a posterior over models using particle-based variational inference with repulsive regularization. By enabling tail-sensitive and class/metaclass-sensitive utilities, RF-DLC improves misprediction risks relevant to real-world tasks while providing calibrated uncertainty estimates. Extensive experiments on standard and real-world data demonstrate improved False Head Rate, robustness across utilities, and strong uncertainty quantification, highlighting practical impact for safety-critical and fairness-aware applications.

Abstract

Long-tailed classification is challenging due to its heavy imbalance in class probabilities. While existing methods often focus on overall accuracy or accuracy for tail classes, they overlook a critical aspect: certain types of errors can carry greater risks than others in real-world long-tailed problems. For example, misclassifying patients (a tail class) as healthy individuals (a head class) entails far more serious consequences than the reverse scenario. To address this critical issue, we introduce Making Reliable and Flexible Decisions in Long-tailed Classification (RF-DLC), a novel framework aimed at reliable predictions in long-tailed problems. Leveraging Bayesian Decision Theory, we introduce an integrated gain to seamlessly combine long-tailed data distributions and the decision-making procedure. We further propose an efficient variational optimization strategy for the decision risk objective. Our method adapts readily to diverse utility matrices, which can be designed for specific tasks, ensuring its flexibility for different problem settings. In empirical evaluation, we design a new metric, False Head Rate, to quantify tail-sensitivity risk, along with comprehensive experiments on multiple real-world tasks, including large-scale image classification and uncertainty quantification, to demonstrate the reliability and flexibility of our method.

Making Reliable and Flexible Decisions in Long-tailed Classification

TL;DR

The paper addresses the challenge of making reliable decisions in long-tailed classification by embedding decision risk into the learning objective via an integrated gain grounded in Bayesian Decision Theory. It introduces a flexible utility-based framework RF-DLC that aligns training with the testing distribution through importance weighting and learns a posterior over models using particle-based variational inference with repulsive regularization. By enabling tail-sensitive and class/metaclass-sensitive utilities, RF-DLC improves misprediction risks relevant to real-world tasks while providing calibrated uncertainty estimates. Extensive experiments on standard and real-world data demonstrate improved False Head Rate, robustness across utilities, and strong uncertainty quantification, highlighting practical impact for safety-critical and fairness-aware applications.

Abstract

Long-tailed classification is challenging due to its heavy imbalance in class probabilities. While existing methods often focus on overall accuracy or accuracy for tail classes, they overlook a critical aspect: certain types of errors can carry greater risks than others in real-world long-tailed problems. For example, misclassifying patients (a tail class) as healthy individuals (a head class) entails far more serious consequences than the reverse scenario. To address this critical issue, we introduce Making Reliable and Flexible Decisions in Long-tailed Classification (RF-DLC), a novel framework aimed at reliable predictions in long-tailed problems. Leveraging Bayesian Decision Theory, we introduce an integrated gain to seamlessly combine long-tailed data distributions and the decision-making procedure. We further propose an efficient variational optimization strategy for the decision risk objective. Our method adapts readily to diverse utility matrices, which can be designed for specific tasks, ensuring its flexibility for different problem settings. In empirical evaluation, we design a new metric, False Head Rate, to quantify tail-sensitivity risk, along with comprehensive experiments on multiple real-world tasks, including large-scale image classification and uncertainty quantification, to demonstrate the reliability and flexibility of our method.
Paper Structure (49 sections, 25 equations, 3 figures, 16 tables, 1 algorithm)

This paper contains 49 sections, 25 equations, 3 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Long-tailed Classification.
  4. Bayesian Decision Theory and Cost-sensitive Learning.
  5. Background
  6. Long-tailed Distributions.
  7. Bayesian Decision Theory.
  8. Methodology
  9. How to Design the Decision Gain?
  10. Standard Classification.
  11. Tail-sensitive Classification.
  12. Class/Meta-class-sensitive Classification.
  13. How to handle the Distribution Shift?
  14. How to Learn the Posterior Distribution?
  15. Relationship with Standard Variational Inference.
  16. ...and 34 more sections

Figures (3)

  • Figure 1: Examples of utility matrices, designed for (a) standard and (b) tail-sensitive classifications, along with (c) class-sensitive classification for bird-plane misprediction and (d) meta-class-sensitive classification for mammal-vehicle misprediction. The entries in the matrix reflect the risk levels ($-1$ is the most risky) and can be flexibly assigned based on task requirements.
  • Figure 2: Ablation studies on CIFAR100-LT. Comparing (a) different forms of class probabilities and (b) varying numbers of particles in terms of overall accuracy and the accuracy of 3 class regions.
  • Figure 3: Full utility matrix configurations on CIFAR10-LT. The two matrices are designed to particularly avoid certain misprediction types (bird-place and mammal-vehicle mispredictions respectively).

Theorems & Definitions (2)

  • proof
  • proof