Enhancing Few-Shot Out-of-Distribution Detection via the Refinement of Foreground and Background

TL;DR

This work tackles few-shot out-of-distribution (FS-OOD) detection in vision-language systems by refining foreground and background information extracted via CLIP-based FG-BG decomposition. It introduces FoBoR, a plug-and-play framework with two key modules—Adaptive Background Suppression (ABS) and Confusable Foreground Rectification (CFR)—to assign differential importance to background patches and to suppress confusable foreground regions, respectively, while preserving a parameter-free, lightweight design. Empirical results show that FoBoR consistently boosts performance when integrated with FG-BG methods (e.g., LoCoOp, SCT, Mambo) and also improves non-FG-BG baselines on standard ImageNet-1k and hard OpenOOD benchmarks, reducing FPR95 and increasing AUROC. The work demonstrates substantial improvements in both average-case and hard OOD settings, highlighting FoBoR’s practical impact for robust FS-OOD detection in CLIP-based pipelines.

Abstract

CLIP-based foreground-background (FG-BG) decomposition methods have demonstrated remarkable effectiveness in improving few-shot out-of-distribution (OOD) detection performance. However, existing approaches still suffer from several limitations. For background regions obtained from decomposition, existing methods adopt a uniform suppression strategy for all patches, overlooking the varying contributions of different patches to the prediction. For foreground regions, existing methods fail to adequately consider that some local patches may exhibit appearance or semantic similarity to other classes, which may mislead the training process. To address these issues, we propose a new plug-and-play framework. This framework consists of three core components: (1) a Foreground-Background Decomposition module, which follows previous FG-BG methods to separate an image into foreground and background regions; (2) an Adaptive Background Suppression module, which adaptively weights patch classification entropy; and (3) a Confusable Foreground Rectification module, which identifies and rectifies confusable foreground patches. Extensive experimental results demonstrate that the proposed plug-and-play framework significantly improves the performance of existing FG-BG decomposition methods. Code is available at: https://github.com/lounwb/FoBoR.

Figures (2)

• Figure 1: The proposed FoBoR framework consists of three components, i.e.,Foreground-Background Decomposition (light beige), Adaptive Background Suppression (light green), and Confusable Foreground Rectification (light blue). The foreground-background decomposition module follows LoCoOp miyai2023locoop by computing the similarity between class text features $\boldsymbol{g}$ and local image features $\{f_i\}_{i=1}^{N}$, for each image patch, and ranking these similarities to partition the input image into foreground and background regions. For background patches, the adaptive background suppression module is proposed to compute local-to-global attention scores using the background query vectors $\{q_i\}_{i=1}^{N}$ and the global key vector $v$ output by CLIP, and to calibrate these scores with the ground-truth probability $p(y|x)$ so as to weight the classification entropy of background patches. For foreground patches, the confusable foreground rectification module is introduced to fuse multi-modal similarities to select confusable classes, and further localize confusable foreground patches based on local similarities, finally maximizing the binary classification entropy between the ground-truth class and the confusable classes on foreground patches.
• Figure 2: Illustration of background-class correlation and its impact on model predictions. The left side shows training samples of the "dog" class under different background conditions. The right side presents the corresponding prediction results when other images are placed under grassland and airplane backgrounds, respectively.