Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Bayesian Confidence (BACON) Estimator for Deep Neural Networks

Patrick D. Kee, Max J. Brown, Jonathan C. Rice, Christian A. Howell

TL;DR

The BACON estimator provides superior ECE and ACE calibration error compared to Softmax for ResNet-18 at 85% network accuracy, and EfficientNet-B0 at 95% network accuracy, on the CIFAR-10 dataset with an imbalanced test set, except for very high accuracy edge cases.

Abstract

This paper introduces the Bayesian Confidence Estimator (BACON) for deep neural networks. Current practice of interpreting Softmax values in the output layer as probabilities of outcomes is prone to extreme predictions of class probability. In this work we extend Waagen's method of representing the terminal layers with a geometric model, where the probability associated with an output vector is estimated with Bayes' Rule using validation data to provide likelihood and normalization values. This estimator provides superior ECE and ACE calibration error compared to Softmax for ResNet-18 at 85% network accuracy, and EfficientNet-B0 at 95% network accuracy, on the CIFAR-10 dataset with an imbalanced test set, except for very high accuracy edge cases. In addition, when using the ACE metric, BACON demonstrated improved calibration error when estimating probabilities for the imbalanced test set when using actual class distribution fractions.

The Bayesian Confidence (BACON) Estimator for Deep Neural Networks

TL;DR

The BACON estimator provides superior ECE and ACE calibration error compared to Softmax for ResNet-18 at 85% network accuracy, and EfficientNet-B0 at 95% network accuracy, on the CIFAR-10 dataset with an imbalanced test set, except for very high accuracy edge cases.

Abstract

This paper introduces the Bayesian Confidence Estimator (BACON) for deep neural networks. Current practice of interpreting Softmax values in the output layer as probabilities of outcomes is prone to extreme predictions of class probability. In this work we extend Waagen's method of representing the terminal layers with a geometric model, where the probability associated with an output vector is estimated with Bayes' Rule using validation data to provide likelihood and normalization values. This estimator provides superior ECE and ACE calibration error compared to Softmax for ResNet-18 at 85% network accuracy, and EfficientNet-B0 at 95% network accuracy, on the CIFAR-10 dataset with an imbalanced test set, except for very high accuracy edge cases. In addition, when using the ACE metric, BACON demonstrated improved calibration error when estimating probabilities for the imbalanced test set when using actual class distribution fractions.

Paper Structure

This paper contains 13 sections, 8 equations, 15 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Material and Methods
  3. Experimental
  4. Theory/Calculation
  5. Results
  6. Entire Dataset Metrics
  7. Expected Calibration Error
  8. Adaptive Calibration Error
  9. Maximum Confidence Error
  10. Class-by-class Metrics
  11. Discussion
  12. Conclusion
  13. Acknowledgements

Figures (15)

  • Figure 1: Validation Dataset Confusion Matrices
  • Figure 2: Geometric interpretation of the relationship between decision vector and output layer weight vectors
  • Figure 3: Output node angle distributions by label class for the same output node.
  • Figure 4: Probability estimation from probability density function
  • Figure 5: Softmax Reliability Diagram for EfficientNet-B0 at 95% accuracy
  • ...and 10 more figures