Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Combining Statistical Depth and Fermat Distance for Uncertainty Quantification

Hai-Vy Nguyen, Fabrice Gamboa, Reda Chhaibi, Sixin Zhang, Serge Gratton, Thierry Giaccone

TL;DR

This work tackles Out-of-domain uncertainty in neural networks by applying a non-parametric framework based on Lens Depth (LD) and Fermat Distance to the feature space, enabling a test-time uncertainty score $S(x)$ without retraining. It introduces a Modified Sample Fermat Distance to fix artifacts and demonstrates stability and effectiveness on toy datasets and real benchmarks (FashionMNIST/MNIST and CIFAR10/SVHN) with favorable comparisons to Gaussian-based, distance-based, and ensemble methods. The approach is non-parametric and non-intrusive, scalable to class- and cluster-wise LDs, and leverages per-class LDs via a max operation to form robust OOD uncertainty scores. Overall, the method provides a principled, geometry- and density-aware uncertainty quantification that complements existing training-time techniques and can extend to kernel-based settings.

Abstract

We measure the Out-of-domain uncertainty in the prediction of Neural Networks using a statistical notion called ``Lens Depth'' (LD) combined with Fermat Distance, which is able to capture precisely the ``depth'' of a point with respect to a distribution in feature space, without any assumption about the form of distribution. Our method has no trainable parameter. The method is applicable to any classification model as it is applied directly in feature space at test time and does not intervene in training process. As such, it does not impact the performance of the original model. The proposed method gives excellent qualitative result on toy datasets and can give competitive or better uncertainty estimation on standard deep learning datasets compared to strong baseline methods.

Combining Statistical Depth and Fermat Distance for Uncertainty Quantification

TL;DR

This work tackles Out-of-domain uncertainty in neural networks by applying a non-parametric framework based on Lens Depth (LD) and Fermat Distance to the feature space, enabling a test-time uncertainty score without retraining. It introduces a Modified Sample Fermat Distance to fix artifacts and demonstrates stability and effectiveness on toy datasets and real benchmarks (FashionMNIST/MNIST and CIFAR10/SVHN) with favorable comparisons to Gaussian-based, distance-based, and ensemble methods. The approach is non-parametric and non-intrusive, scalable to class- and cluster-wise LDs, and leverages per-class LDs via a max operation to form robust OOD uncertainty scores. Overall, the method provides a principled, geometry- and density-aware uncertainty quantification that complements existing training-time techniques and can extend to kernel-based settings.

Abstract

We measure the Out-of-domain uncertainty in the prediction of Neural Networks using a statistical notion called ``Lens Depth'' (LD) combined with Fermat Distance, which is able to capture precisely the ``depth'' of a point with respect to a distribution in feature space, without any assumption about the form of distribution. Our method has no trainable parameter. The method is applicable to any classification model as it is applied directly in feature space at test time and does not intervene in training process. As such, it does not impact the performance of the original model. The proposed method gives excellent qualitative result on toy datasets and can give competitive or better uncertainty estimation on standard deep learning datasets compared to strong baseline methods.
Paper Structure (28 sections, 1 theorem, 8 equations, 12 figures, 3 tables)

This paper contains 28 sections, 1 theorem, 8 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Lens Depth
  5. Fermat distance
  6. Combining LD and Fermat Distance
  7. Artifacts from classical Fermat distance
  8. Modified Sample Fermat Distance
  9. Qualitative evaluation of stability
  10. Stability with respect to number of training points
  11. Stability with respect to the hyperparameter $\alpha$
  12. From LD to OOD uncertainty score
  13. Reducing complexity for computing $LD$
  14. Experiments on Neural Networks
  15. How is the input distribution represented in the feature space of softmax model?
  16. ...and 13 more sections

Key Result

Proposition 1

For $x\in\mathbb{R}^d, \;\widehat{LD}(x)=\widehat{LD}(q_Q(x))$. In other words, the empirical lens depth is constant over the Voronoï cellsDefinition of the Voronoï cells is in Appendixvoronoi. associated to $Q$.

Figures (12)

  • Figure 1.1: Motivation example using the two-moons dataset. Colors indicate OOD score. Gaussian fitting (left), fails completely to capture the distribution of dataset whereas our proposed method (right) represents very well how central a point is with respect to (w.r.t.) clusters without any prior assumption.
  • Figure 1.2: General scheme of our method. Given a set of features $\Phi$, the Fermat distance is a metric which respects and adapts to the distribution of $\Phi$. Lens depth wraps the Fermat distance into a probabilistic and interpretable score $S$. No additional training is needed.
  • Figure 2.1: $LD$ using only $20\%$ of points (200 points) on simulated spiral dataset of 1000 points over 10 runs. We see that the contours of $LD$ level changes slightly between different tries, but in general, the proposed method captures well the general form of distribution. Note that the points presented in the plot are the full dataset of 1000 points.
  • Figure 3.1: $\widehat{LD}$ using Euclidean distance. We see that using simply Euclidean distance cannot capture correctly the distribution.
  • Figure 3.2: Sample Fermat path between two fixed randomly chosen points using different values of $\alpha$.
  • ...and 7 more figures

Theorems & Definitions (2)

  • Proposition 1
  • proof