Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging OOD Detection and Generalization: A Graph-Theoretic View

Han Wang, Yixuan Li

TL;DR

A graph-theoretic framework is introduced to jointly tackle both OOD generalization and detection problems, and data representations are obtained through the factorization of the graph's adjacency matrix, enabling us to derive provable error quantifying OOD generalization and detection performance.

Abstract

In the context of modern machine learning, models deployed in real-world scenarios often encounter diverse data shifts like covariate and semantic shifts, leading to challenges in both out-of-distribution (OOD) generalization and detection. Despite considerable attention to these issues separately, a unified framework for theoretical understanding and practical usage is lacking. To bridge the gap, we introduce a graph-theoretic framework to jointly tackle both OOD generalization and detection problems. By leveraging the graph formulation, data representations are obtained through the factorization of the graph's adjacency matrix, enabling us to derive provable error quantifying OOD generalization and detection performance. Empirical results showcase competitive performance in comparison to existing methods, thereby validating our theoretical underpinnings. Code is publicly available at https://github.com/deeplearning-wisc/graph-spectral-ood.

Bridging OOD Detection and Generalization: A Graph-Theoretic View

TL;DR

A graph-theoretic framework is introduced to jointly tackle both OOD generalization and detection problems, and data representations are obtained through the factorization of the graph's adjacency matrix, enabling us to derive provable error quantifying OOD generalization and detection performance.

Abstract

In the context of modern machine learning, models deployed in real-world scenarios often encounter diverse data shifts like covariate and semantic shifts, leading to challenges in both out-of-distribution (OOD) generalization and detection. Despite considerable attention to these issues separately, a unified framework for theoretical understanding and practical usage is lacking. To bridge the gap, we introduce a graph-theoretic framework to jointly tackle both OOD generalization and detection problems. By leveraging the graph formulation, data representations are obtained through the factorization of the graph's adjacency matrix, enabling us to derive provable error quantifying OOD generalization and detection performance. Empirical results showcase competitive performance in comparison to existing methods, thereby validating our theoretical underpinnings. Code is publicly available at https://github.com/deeplearning-wisc/graph-spectral-ood.
Paper Structure (40 sections, 7 theorems, 60 equations, 6 figures, 9 tables)

This paper contains 40 sections, 7 theorems, 60 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Graph-Based Framework for OOD Generalization and Detection
  4. Graph Formulation
  5. Learning Representations Based on Graph Spectral
  6. A surrogate objective.
  7. Theoretical Analysis
  8. Analytic Form of Learned Representations
  9. Analysis Target
  10. An Illustrative Example
  11. Main analysis.
  12. Interpretation.
  13. Experiments
  14. Experimental Setup
  15. Implementation details.
  16. ...and 25 more sections

Key Result

Theorem 3.3

We define each row $\*f_x^{\top}$ of $F$ as a scaled version of learned feature embedding $f:\mathcal{X}\mapsto \mathbb{R}^k$, with $\*f_x = \sqrt{w_x}f(x)$. Then minimizing the loss function $\mathcal{L}_\text{mf}(F, A)$ in Equation eq:lmf is equivalent to minimizing the surrogate loss in Equation

Figures (6)

  • Figure 1: Illustration of our graph-theoretic framework for joint out-of-distribution generalization and detection. Left: Graph formulation containing three types of data in the wild: ID (e.g., seabird), covariate OOD (e.g., bird in the forest), and semantic OOD (e.g., dog). Right: Graph factorization for obtaining the closed-form solution of the data representations, which are used to derive OOD generalization and OOD detection errors.
  • Figure 2: Illustration of graph and augmentation probability.
  • Figure 3: Value of function $S(f)$
  • Figure 4: (a) Distribution of KNN distance. (b) t-SNE visualization of learned embeddings. We employ CIFAR-10 as $\mathbb{P}_{\text{in}}$, CIFAR-10-C as $\mathbb{P}_{\text{out}}^{\text{covariate}}$, and SVHN as $\mathbb{P}_{\text{out}}^{\text{semantic}}$.
  • Figure 5: Illustration of 5 nodes graph and the augmentation probability defined by classes and domains. Figure (a) illustrates the scenario where semantic OOD data has a different domain from covariate OOD. Figure (b) depicts the case where semantic OOD and covariate OOD share the same domain.
  • ...and 1 more figures

Theorems & Definitions (12)

  • Definition 2.1
  • Definition 2.2
  • Definition 3.1: Self-supervised connectivity
  • Definition 3.2: Total edge connectivity
  • Theorem 3.3: Theoretical equivalence between two objectives
  • Theorem 4.1
  • Theorem 4.2
  • proof
  • Theorem B.1
  • Theorem B.2
  • ...and 2 more