Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Selective Learning: Towards Robust Calibration with Dynamic Regularization

Zongbo Han, Yifeng Yang, Changqing Zhang, Linjun Zhang, Joey Tianyi Zhou, Qinghua Hu

TL;DR

This work tackles miscalibration in deep classifiers by addressing the conflicting goals of accuracy and entropy-based regularization. It introduces Dynamic Regularization (DReg), which models training data as a mix of in-distribution and out-of-capability samples via the $\eta$-Huber contamination model and applies regularization selectively to out-of-capability samples, while allowing easy samples to be learned with high confidence. Theoretically, DReg achieves a smaller calibration error than traditional regularization-based methods under contamination, and empirically it delivers robust calibration and strong accuracy across diverse datasets and augmentation regimes. The method leverages challenging samples in the training data to provide explicit confidence supervision, improving reliability of predictions in real-world, distribution-shifted, or outlier-prone settings. Overall, DReg offers a principled, outlier-aware approach to calibrating neural networks with practical implications for safety-critical applications.

Abstract

Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. This problem usually arises due to the overfitting problem, which is characterized by learning everything presented in the training set, resulting in overconfident predictions during testing. Existing methods typically address overfitting and mitigate the miscalibration by adding a maximum-entropy regularizer to the objective function. The objective can be understood as seeking a model that fits the ground-truth labels by increasing the confidence while also maximizing the entropy of predicted probabilities by decreasing the confidence. However, previous methods lack clear guidance on confidence adjustment, leading to conflicting objectives (increasing but also decreasing confidence). Therefore, we introduce a method called Dynamic Regularization (DReg), which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off. At a high level, DReg aims to obtain a more reliable model capable of acknowledging what it knows and does not know. Specifically, DReg effectively fits the labels for in-distribution samples (samples that should be learned) while applying regularization dynamically to samples beyond model capabilities (e.g., outliers), thereby obtaining a robust calibrated model especially on the samples beyond model capabilities. Both theoretical and empirical analyses sufficiently demonstrate the superiority of DReg compared with previous methods.

Selective Learning: Towards Robust Calibration with Dynamic Regularization

TL;DR

This work tackles miscalibration in deep classifiers by addressing the conflicting goals of accuracy and entropy-based regularization. It introduces Dynamic Regularization (DReg), which models training data as a mix of in-distribution and out-of-capability samples via the -Huber contamination model and applies regularization selectively to out-of-capability samples, while allowing easy samples to be learned with high confidence. Theoretically, DReg achieves a smaller calibration error than traditional regularization-based methods under contamination, and empirically it delivers robust calibration and strong accuracy across diverse datasets and augmentation regimes. The method leverages challenging samples in the training data to provide explicit confidence supervision, improving reliability of predictions in real-world, distribution-shifted, or outlier-prone settings. Overall, DReg offers a principled, outlier-aware approach to calibrating neural networks with practical implications for safety-critical applications.

Abstract

Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. This problem usually arises due to the overfitting problem, which is characterized by learning everything presented in the training set, resulting in overconfident predictions during testing. Existing methods typically address overfitting and mitigate the miscalibration by adding a maximum-entropy regularizer to the objective function. The objective can be understood as seeking a model that fits the ground-truth labels by increasing the confidence while also maximizing the entropy of predicted probabilities by decreasing the confidence. However, previous methods lack clear guidance on confidence adjustment, leading to conflicting objectives (increasing but also decreasing confidence). Therefore, we introduce a method called Dynamic Regularization (DReg), which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off. At a high level, DReg aims to obtain a more reliable model capable of acknowledging what it knows and does not know. Specifically, DReg effectively fits the labels for in-distribution samples (samples that should be learned) while applying regularization dynamically to samples beyond model capabilities (e.g., outliers), thereby obtaining a robust calibrated model especially on the samples beyond model capabilities. Both theoretical and empirical analyses sufficiently demonstrate the superiority of DReg compared with previous methods.
Paper Structure (11 sections, 1 theorem, 19 equations, 5 figures, 2 tables)

This paper contains 11 sections, 1 theorem, 19 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dynamic Regularization for Calibration
  4. Problem Setting
  5. Regularization for Calibration
  6. Dynamic Regularization
  7. Theory
  8. Experiments
  9. Experimental Setup
  10. Experimental Results
  11. Conclusion

Key Result

Theorem 3.1

Consider the data generative model and the learning setting above. We assume $\|\boldsymbol{w}^*\|\le c_0$ for some sufficiently small $c_0>0$, and $d/n=o(1)$. Suppose the initialization parameter $\boldsymbol{\theta}^{(0)}$ satisfies $\|\boldsymbol{\theta}^{(0)}-\boldsymbol{w}^*\|\le c_1$ for a suf

Figures (5)

  • Figure 1: The motivation of the DReg. Previous regularization-based methods aim for accurate classification while maximizing predictive entropy, resulting in conflicting optimization goals of simultaneously increasing and decreasing confidence. To address this issue, DReg dynamically applies regularization to avoid these conflicting optimization objectives, assigning higher confidence to easy samples and lower confidence to challenging samples.
  • Figure 2: Performance of the model on multiple metrics with varying fraction hyperparameter $\eta$, while fixing the regularization strength $\beta$ to 1, on the CIFAR-8-2 dataset.
  • Figure 3: Performance of the model on different metrics with varying regularization strength hyperparameter $\beta$, while fixing the outlier fraction hyperparameter $\eta$ to 0.2, on the CIFAR-8-2 dataset.
  • Figure 4: Performance of the model on multiple metrics with varying challenging samples fraction hyperparameter $\eta$, while fixing the regularization strength $\beta$ to 1, on the CIFAR-80-20 dataset.
  • Figure 5: Performance of the model on different metrics with varying regularization strength hyperparameter $\beta$, while fixing the challenging samples fraction hyperparameter $\eta$ to 0.2, on the CIFAR-80-20 dataset.

Theorems & Definitions (2)

  • Theorem 3.1
  • proof