Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Intermediate Representations for Better Out-of-Distribution Detection

Gianluca Guglielmo, Marc Masana

TL;DR

The paper addresses the challenge of reliable out-of-distribution (OoD) detection without sacrificing in-distribution (ID) performance. It investigates intermediate-layer activations and proposes two methods: Ag-EBO, which aggregates per-layer energies for layer-agnostic OoD scoring, and R-EBO, which regularizes hidden layers with an energy-based loss to promote discriminative intermediate representations. Energies are defined per layer as $E_l({\mathbf{x}}) = -T \log \sum_{i} e^{a_l^{i}({\mathbf{x}})/T}$ and combined into $\mathbf{E}({\mathbf{x}})=(E_1({\mathbf{x}}),\dots,E_L({\mathbf{x}}))$, with evaluations across architectures and datasets (OpenOOD benchmarks) showing improved OoD detection in many settings, albeit sometimes at a small cost to ID accuracy. The work highlights practical implications for real-time OoD detection and outlines limitations related to layer selection and generalization, guiding future directions such as ID-only regularization and synthetic-data augmentation.

Abstract

In real-world applications, machine learning models must reliably detect Out-of-Distribution (OoD) samples to prevent unsafe decisions. Current OoD detection methods often rely on analyzing the logits or the embeddings of the penultimate layer of a neural network. However, little work has been conducted on the exploitation of the rich information encoded in intermediate layers. To address this, we analyze the discriminative power of intermediate layers and show that they can positively be used for OoD detection. Therefore, we propose to regularize intermediate layers with an energy-based contrastive loss, and by grouping multiple layers in a single aggregated response. We demonstrate that intermediate layer activations improves OoD detection performance by running a comprehensive evaluation across multiple datasets.

Leveraging Intermediate Representations for Better Out-of-Distribution Detection

TL;DR

The paper addresses the challenge of reliable out-of-distribution (OoD) detection without sacrificing in-distribution (ID) performance. It investigates intermediate-layer activations and proposes two methods: Ag-EBO, which aggregates per-layer energies for layer-agnostic OoD scoring, and R-EBO, which regularizes hidden layers with an energy-based loss to promote discriminative intermediate representations. Energies are defined per layer as and combined into , with evaluations across architectures and datasets (OpenOOD benchmarks) showing improved OoD detection in many settings, albeit sometimes at a small cost to ID accuracy. The work highlights practical implications for real-time OoD detection and outlines limitations related to layer selection and generalization, guiding future directions such as ID-only regularization and synthetic-data augmentation.

Abstract

In real-world applications, machine learning models must reliably detect Out-of-Distribution (OoD) samples to prevent unsafe decisions. Current OoD detection methods often rely on analyzing the logits or the embeddings of the penultimate layer of a neural network. However, little work has been conducted on the exploitation of the rich information encoded in intermediate layers. To address this, we analyze the discriminative power of intermediate layers and show that they can positively be used for OoD detection. Therefore, we propose to regularize intermediate layers with an energy-based contrastive loss, and by grouping multiple layers in a single aggregated response. We demonstrate that intermediate layer activations improves OoD detection performance by running a comprehensive evaluation across multiple datasets.

Paper Structure

This paper contains 14 sections, 11 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Out-of-Distribution Detection
  4. Problem statement
  5. Energy-based out-of-distribution detection.
  6. OoD with Intermediate Layers
  7. Motivation
  8. Energy aggregation (Ag-EBO)
  9. Energy regularization (R-EBO)
  10. Experimental results
  11. Implementation details
  12. Analysis of OoD with intermediate layers
  13. Analysis on different architectures
  14. Discussion and Limitations

Figures (3)

  • Figure 1: Intermediate representations are often more informative than the logits when dealing with OoD detection.
  • Figure 2: AUROC scores for OoD detection for each intermediate layer of ResNet18 are presented. The network is pretrained on CIFAR-10 ($\mathcal{D}_{\text{in}}$) and evaluated against the corresponding $\mathcal{D}_{out}^{near}$ and $\mathcal{D}_{out}^{far}$ datasets. Results are averaged across datasets in both categories.
  • Figure 3: AUROC scores for each intermediate layer of ResNet18 pretrained on CIFAR-10 as $\mathcal{D}_{in}$ and evaluated against different corruptions (CIFAR-10-C). Results are averaged over all corruption types and seeds.