Towards Few-shot Out-of-Distribution Detection

TL;DR

This paper targets the challenge of detecting out-of-distribution (OOD) samples when only a few labeled in-distribution examples are available. It introduces a comprehensive FS-OOD detection benchmark and shows that parameter-efficient fine-tuning (PEFT) methods, such as visual prompt tuning (VPT) and visual adapter tuning (VAT), outperform full fine-tuning and linear probing in the few-shot regime. The authors propose Domain-Specific and General Knowledge Fusion (DSGF), which concatenates features from the original pre-trained model with those from the fine-tuned model to preserve general knowledge while capturing domain-specific adaptations; DSGF yields substantial gains across tuning paradigms and OOD detectors, especially in few-shot settings, and improves ID accuracy as well. The work emphasizes the practical impact of preserving pre-trained general knowledge for OOD detection and offers a versatile, model-agnostic fusion approach with publicly shareable code.

Abstract

Out-of-distribution (OOD) detection is critical for ensuring the reliability of open-world intelligent systems. Despite the notable advancements in existing OOD detection methodologies, our study identifies a significant performance drop under the scarcity of training samples. In this context, we introduce a novel few-shot OOD detection benchmark, carefully constructed to address this gap. Our empirical analysis reveals the superiority of ParameterEfficient Fine-Tuning (PEFT) strategies, such as visual prompt tuning and visual adapter tuning, over conventional techniques, including fully fine-tuning and linear probing tuning in the few-shot OOD detection task. Recognizing some crucial information from the pre-trained model, which is pivotal for OOD detection, may be lost during the fine-tuning process, we propose a method termed DomainSpecific and General Knowledge Fusion (DSGF). This approach is designed to be compatible with diverse fine-tuning frameworks. Our experiments show that the integration of DSGF significantly enhances the few-shot OOD detection capabilities across various methods and fine-tuning methodologies, including fully fine-tuning, visual adapter tuning, and visual prompt tuning. The code will be released.

Figures (8)

• Figure 1: Comparison of different OOD detection methods in the FS-OOD detection task. Our DSGF significantly improves the performance of all baseline methods. 'Avg.' represents the average of six OOD detection baseline methods, and ' + DSGF' denotes deploying our method.
• Figure 2: We include FFT, LPT, VAT, and VPT in our few-shot OOD detection benchmark (Stage 1). Our DSFG method fuses the general feature of the pre-trained model and the domain-specific feature of the fine-tuned model for better OOD detection performance (Stage 2).
• Figure 3: Comparison of AUROC scores between without and with fully fine-tuning for three ID datasets. OOD Scores are computed by the k-NN method.
• Figure 4: Main Results of FS-OOD detection on different tuning paradigms. Overall, our method achieves more performance gains in few-shot settings. 'Avg.' represents the arithmetic average of six OOD score evaluation methods. 'DSGF' denotes deploying our method.
• Figure 5: The t-sne visualization on Imagenet-1k (ID) and iNaturalist (OOD). We use the output features from the last layer of the Transformer to visualize the distribution of ID and OOD samples.vspace-0.5cm