Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TULiP: Test-time Uncertainty Estimation via Linearization and Weight Perturbation

Yuhui Zhang, Dongshen Wu, Yuichiro Wada, Takafumi Kanamori

TL;DR

TULiP introduces a theoretically grounded, post-hoc uncertainty estimator for OOD detection by analyzing perturbations in the linearized training dynamics via the Neural Tangent Kernel. By hypothetically perturbing the network before convergence and bounding the resulting fluctuations, it constructs a surrogate posterior ensemble whose predictions are combined to yield robust OOD scores, notably enhancing near-distribution detection without access to training data. The method is implemented through practical steps to estimate Jacobians, calibrate kernel-related quantities, and build a Surrogate Posterior Envelope (SPE), with empirical validation on OpenOOD benchmarks showing state-of-the-art or competitive performance across near and far OOD settings. The work demonstrates a principled link between training dynamics and inference-time uncertainty, offering a scalable, plug-and-play tool that improves existing post-hoc detectors and opens directions for extending the framework to broader learning paradigms.

Abstract

A reliable uncertainty estimation method is the foundation of many modern out-of-distribution (OOD) detectors, which are critical for safe deployments of deep learning models in the open world. In this work, we propose TULiP, a theoretically-driven post-hoc uncertainty estimator for OOD detection. Our approach considers a hypothetical perturbation applied to the network before convergence. Based on linearized training dynamics, we bound the effect of such perturbation, resulting in an uncertainty score computable by perturbing model parameters. Ultimately, our approach computes uncertainty from a set of sampled predictions. We visualize our bound on synthetic regression and classification datasets. Furthermore, we demonstrate the effectiveness of TULiP using large-scale OOD detection benchmarks for image classification. Our method exhibits state-of-the-art performance, particularly for near-distribution samples.

TULiP: Test-time Uncertainty Estimation via Linearization and Weight Perturbation

TL;DR

TULiP introduces a theoretically grounded, post-hoc uncertainty estimator for OOD detection by analyzing perturbations in the linearized training dynamics via the Neural Tangent Kernel. By hypothetically perturbing the network before convergence and bounding the resulting fluctuations, it constructs a surrogate posterior ensemble whose predictions are combined to yield robust OOD scores, notably enhancing near-distribution detection without access to training data. The method is implemented through practical steps to estimate Jacobians, calibrate kernel-related quantities, and build a Surrogate Posterior Envelope (SPE), with empirical validation on OpenOOD benchmarks showing state-of-the-art or competitive performance across near and far OOD settings. The work demonstrates a principled link between training dynamics and inference-time uncertainty, offering a scalable, plug-and-play tool that improves existing post-hoc detectors and opens directions for extending the framework to broader learning paradigms.

Abstract

A reliable uncertainty estimation method is the foundation of many modern out-of-distribution (OOD) detectors, which are critical for safe deployments of deep learning models in the open world. In this work, we propose TULiP, a theoretically-driven post-hoc uncertainty estimator for OOD detection. Our approach considers a hypothetical perturbation applied to the network before convergence. Based on linearized training dynamics, we bound the effect of such perturbation, resulting in an uncertainty score computable by perturbing model parameters. Ultimately, our approach computes uncertainty from a set of sampled predictions. We visualize our bound on synthetic regression and classification datasets. Furthermore, we demonstrate the effectiveness of TULiP using large-scale OOD detection benchmarks for image classification. Our method exhibits state-of-the-art performance, particularly for near-distribution samples.

Paper Structure

This paper contains 64 sections, 6 theorems, 50 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Uncertainty Quantification (UQ)
  4. Post-hoc OOD Detectors
  5. Theoretical Framework
  6. Preliminaries: Linearized Training Dynamics
  7. Notations
  8. Modeling Uncertainty
  9. Bounding Linearized Training Fluctuations
  10. Network Ensemble
  11. Implementation
  12. Estimation of Jacobian (Line 2 - 8)
  13. Calibration on Validation Dataset
  14. Surrogate Posterior Envelope (SPE) (Line 9 - 13)
  15. Experiments
  16. ...and 49 more sections

Key Result

theorem thmcountertheorem

Under assumptions A1-A4, for a network $f$ trained with Eq. eq: Linearized dynamics and a perturbed network $\hat{f}$ trained with Eq. eq: Linearized dynamics perturbed, the perturbation applied at time $t_s = T - \Delta T$ bounded by $\alpha$, we have where $C = \frac{\alpha \eta \bar{\Theta}_X^{1/2}}{\lambda_{max}}\left( e^{(T - t_s)L\lambda_{max}} - 1 \right)$, $\bar{\Theta}_X^{1/2} := \left\|

Figures (4)

  • Figure 1: Illustration of the hypothetical perturbation process. The shaded area indicates the distribution of $\hat{f}_T$, and the dashed circle is the Surrogate Posterior Envelope (SPE). See Sec. \ref{['subsec: surrogate-ensemble']} for details.
  • Figure 2: Verification of Thm. \ref{['theorem: direct gronwall']} (Sec. \ref{['subsec: empirical-validation']}). From left to right, (a): Regression on Splines. Light shade: the bound Eq. \ref{['eq: original uncertainty bound']}, heavy shade: Ground-truth ensemble (Eq. \ref{['eq: GT-uncertainty']}), black dots: training data. (b): OOD Detection capability of Eq. \ref{['eq: bound-practical']} on ImageNet-1K setup, using $S$ in Alg. \ref{['alg: TULiP']} (line 13, used to estimate Eq. \ref{['eq: bound-practical']}) as OOD scoring function. (c): Verification of A5 on ImageNet-1K setup.
  • Figure 3: (a) Visualization of $S$ in Alg. \ref{['alg: TULiP']} (line 13, used to estimate Eq. \ref{['eq: bound-practical']}) in various datasets (left to right: ID (ImageNet-1K), NINCO (near), Textures (far)). (b) Visualization of the MSP score before and after applying TULiP on ID (right) and OOD (left). Notice how OOD data is pushed further towards 0 (lower confidence) than ID. We chose MSP due to its simplicity. (c) Effect of TULiP on various OOD scoring criteria for near-OOD (solid) and far-OOD (dashed). $J_\mathrm{scaling}$ (horizontal axis) controls the strength of TULiP, where a value of $0$ indicates baselines without TULiP.
  • Figure 4: Results by varying $\epsilon$ and $\lambda$ on ImageNet-1K ID. Solid lines: A validation set is used to find the the optimal $J$ with $J_\mathrm{scaling} = 2.0$. Dashed lines: No validation set calibration for $J$ (results for $\epsilon = 0.0075$ falls out of the range of the plot).

Theorems & Definitions (12)

  • theorem thmcountertheorem
  • proof
  • proposition thmcounterproposition
  • proof
  • lemma thmcounterlemma
  • theorem thmcountertheorem
  • proof
  • remark thmcounterremark
  • lemma thmcounterlemma
  • proof
  • ...and 2 more