Domain Feature Collapse: Implications for Out-of-Distribution Detection and Solutions

TL;DR

This work identifies a fundamental limitation of supervised learning on single-domain data: domain information is collapsed in learned representations, defined as $I({\mathbf{x}}_{\mathbf{d}}; {\mathbf{z}})=0$ under information bottleneck optimization. The authors formalize this domain feature collapse and extend it with Fano’s inequality to bound partial collapse in practice. They validate the theory with Domain Bench, a suite of single-domain datasets, and demonstrate that preserving domain information via a two-stage Domain Filtering approach—utilizing pretrained domain features for filtering and supervised features for class-based OOD detection—resolves the failure mode and improves OOD robustness across distant and adjacent OOD scenarios. The results motivate a paradigm shift from chasing stronger OOD detectors to designing representation spaces that retain domain information, with practical implications for transfer learning, fine-tuning, and when to freeze pretrained models. The work contributes a theoretical explanation, a benchmark, and a scalable, architecture-driven solution that yields substantial empirical gains across diverse single-domain domains.

Abstract

Why do state-of-the-art OOD detection methods exhibit catastrophic failure when models are trained on single-domain datasets? We provide the first theoretical explanation for this phenomenon through the lens of information theory. We prove that supervised learning on single-domain data inevitably produces domain feature collapse -- representations where I(x_d; z) = 0, meaning domain-specific information is completely discarded. This is a fundamental consequence of information bottleneck optimization: models trained on single domains (e.g., medical images) learn to rely solely on class-specific features while discarding domain features, leading to catastrophic failure when detecting out-of-domain samples (e.g., achieving only 53% FPR@95 on MNIST). We extend our analysis using Fano's inequality to quantify partial collapse in practical scenarios. To validate our theory, we introduce Domain Bench, a benchmark of single-domain datasets, and demonstrate that preserving I(x_d; z) > 0 through domain filtering (using pretrained representations) resolves the failure mode. While domain filtering itself is conceptually straightforward, its effectiveness provides strong empirical evidence for our information-theoretic framework. Our work explains a puzzling empirical phenomenon, reveals fundamental limitations of supervised learning in narrow domains, and has broader implications for transfer learning and when to fine-tune versus freeze pretrained models.

Figures (12)

• Figure 1: Domain Feature Collapse: Supervised learning on single-domain data inevitably produces representations where domain information is lost ($I({\mathbf{x}}_{\mathbf{d}}; {\mathbf{z}}) = 0$). This leads to catastrophic failure in OOD detection, as models cannot distinguish between in-domain and out-of-domain samples without domain-specific features.
• Figure 2: Sample images for the Butterfly dataset.
• Figure 3: Sample images for the Cards dataset.
• Figure 4: Sample images for the Colon dataset.
• Figure 5: Sample images for the Eurosat dataset.

Theorems & Definitions (16)

• Definition 3.1
• Definition 3.2
• Definition 4.1
• Theorem 4.2
• Remark 4.3
• Theorem B.1
• Lemma C.1
• proof
• Lemma C.2
• proof