Recent Advances in Out-of-Distribution Detection with CLIP-Like Models: A Survey

TL;DR

This survey analyzes how CLIP-like vision–language models enable cross-modal out-of-distribution detection, arguing that existing unimodal categorizations miss the multimodal nature of CLIP. It proposes a two-axis categorization—whether OOD images are seen or unseen and whether OOD texts are known or unknown—across train-free and train-required strategies, and then details four corresponding method families. The paper reviews CLIP-based architectures, prompt-learning techniques, and a broad set of OOD detection approaches, comparing their data requirements, textual vs. visual emphasis, and scoring schemes, while highlighting limitations such as information leakage and benchmark biases. It concludes with open problems and future directions, including cross-domain integration, real-world applications, and theoretical analysis of modality gaps, aiming to guide robust, scalable OOD detection in multimodal settings.

Abstract

Out-of-distribution detection (OOD) is a pivotal task for real-world applications that trains models to identify samples that are distributionally different from the in-distribution (ID) data during testing. Recent advances in AI, particularly Vision-Language Models (VLMs) like CLIP, have revolutionized OOD detection by shifting from traditional unimodal image detectors to multimodal image-text detectors. This shift has inspired extensive research; however, existing categorization schemes (e.g., few- or zero-shot types) still rely solely on the availability of ID images, adhering to a unimodal paradigm. To better align with CLIP's cross-modal nature, we propose a new categorization framework rooted in both image and text modalities. Specifically, we categorize existing methods based on how visual and textual information of OOD data is utilized within image + text modalities, and further divide them into four groups: OOD Images (i.e., outliers) Seen or Unseen, and OOD Texts (i.e., learnable vectors or class names) Known or Unknown, across two training strategies (i.e., train-free or training-required). More importantly, we discuss open problems in CLIP-like OOD detection and highlight promising directions for future research, including cross-domain integration, practical applications, and theoretical understanding.

Figures (4)

• Figure 1: Illustration of Out-of-distribution (OOD) Detection. OOD detection is a major computer vision task that addresses semantic distribution shifts between training and testing data, reflecting real-world scenarios. Specifically, during training, the model learns from ID data, as shown in the upper. However, once deployed, the model may encounter both ID (seen) and OOD (unseen) data in the testing phase, as depicted in the below. To ensure reliability, the trained model should not only classify ID samples accurately but also detect OOD instances to avoid incorrect decisions.
• Figure 2: The number of recent CLIP-like OOD detection studies in top venues (up to 02/2025). The abbreviations IU + TK, IU + TU, IS + TU and IS + TK, represent OOD Images Unseen + OOD Texts Known, OOD Images Unseen + OOD Texts Unknown, OOD Images Seen + OOD Texts Unknown and OOD Images Seen + OOD Texts Known, respectively.
• Figure 3: The general pipeline of unimodal and CLIP-like OOD detection paradigms. (a) Unimodal OOD detection models typically use a single-image encoder trained on ID data. The encoder extracts visual embeddings and often employs confidence scores, distance metrics, or density estimation with a threshold. (b) Compared to the unimodal ones, CLIP learns a joint vision-language embedding space by aligning images with textual descriptions. This allows CLIP to map diverse visual concepts into a more semantically structured embedding space.
• Figure 4: An illustration of different CLIP-like OOD detection paradigms, from the perspective of the utilization of OOD visual and textual information. These paradigms are mainly distinguished by whether they require additional OOD image information (represented by gray dashed circles) and whether OOD-specific textual information (represented by gray dashed squares) is incorporated into fine-tuning. In particular, OOD Images Unseen + OOD Texts Unknown is further divided into two groups, Train-free and Train-required (illustrated in Figure (d) with blue descriptions). The primary criterion for this division is whether prompt learning for ID classes is involved.

Theorems & Definitions (1)

• Definition 1: Unimodal Out-of-distribution Detection