Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection

TL;DR

This paper tackles open-world OOD detection in vision-language models under few-shot settings by maximizing local, region-specific information while keeping global prompts fixed. It introduces Local-Prompt, a coarse-to-fine framework with global prompt guided negative augmentation and local prompt enhanced regional regularization, plus a Regional-MCM score to better leverage regional cues. Empirical results on ImageNet-1k show substantial improvements in FPR95 and competitive ID accuracy, with a 5.17 percentage-point reduction in FPR95 under 4-shot tuning and strong near-OOD performance; the approach is compatible with trained global prompts to provide further gains. The method is extensible, supports integration with existing global-prompt strategies, and is validated by ablations and visualizations that confirm the effectiveness of local-region learning for OOD detection.

Abstract

Out-of-Distribution (OOD) detection, aiming to distinguish outliers from known categories, has gained prominence in practical scenarios. Recently, the advent of vision-language models (VLM) has heightened interest in enhancing OOD detection for VLM through few-shot tuning. However, existing methods mainly focus on optimizing global prompts, ignoring refined utilization of local information with regard to outliers. Motivated by this, we freeze global prompts and introduce Local-Prompt, a novel coarse-to-fine tuning paradigm to emphasize regional enhancement with local prompts. Our method comprises two integral components: global prompt guided negative augmentation and local prompt enhanced regional regularization. The former utilizes frozen, coarse global prompts as guiding cues to incorporate negative augmentation, thereby leveraging local outlier knowledge. The latter employs trainable local prompts and a regional regularization to capture local information effectively, aiding in outlier identification. We also propose regional-related metric to empower the enrichment of OOD detection. Moreover, since our approach explores enhancing local prompts only, it can be seamlessly integrated with trained global prompts during inference to boost the performance. Comprehensive experiments demonstrate the effectiveness and potential of our method. Notably, our method reduces average FPR95 by 5.17% against state-of-the-art method in 4-shot tuning on challenging ImageNet-1k dataset, even outperforming 16-shot results of previous methods. Code is released at https://github.com/AuroraZengfh/Local-Prompt.

Figures (9)

• Figure 1: Comparison of prompt learning for OOD detection task. Prompts with global optimization may fail in challenging OOD samples as they are overall similar to ID samples and only have subtle regional differences. For example, cat and tiger are generally similar (blue boxes) and only differ in forehead (red box). Our approach with local outlier knowledge cares about region difference and tackles the issue to some extent.
• Figure 2: Detailed structure of the proposed Local-Prompt. Our method consists of global prompt guided negative augmentation and local prompt enhanced regional regularization. We froze global prompts to select regional augmented samples and enhance local prompts to learn regional-related representation that helps improve both ID accuracy and OOD detection.
• Figure 3: Comparison of density map on iNaturalist. Ours are more separable.
• Figure 4: Ablation study of different OOD score strategies. LNP denotes local negative prompts.
• Figure 5: Visualization of ID and OOD related regions. Local prompts and local negative prompts successfully focus on ID-related and OOD-related regions, respectively, which helps OOD detection.