Reflexive Guidance: Improving OoDD in Vision-Language Models via Self-Guided Image-Adaptive Concept Generation

TL;DR

The paper tackles the reliability gap of vision-language foundation models in out-of-distribution detection (OoDD) by systemically evaluating proprietary and open-source LVLMs and introducing Reflexive Guidance (ReGuide), a two-stage, image-adaptive prompting strategy. ReGuide prompts the LVLM to generate auxiliary OoD concepts from the input image and then uses these concepts as auxiliary OoD classes to improve OoDD through a maximum-confidence scoring scheme $S_{\,max}$. Across ImageNet200 and CIFAR10 benchmarks, ReGuide enhances OoDD performance for both open-source and proprietary models, sometimes boosting in-distribution classification, while revealing calibration and failure-mode challenges in current LVLMs. The work offers a practical, model-agnostic approach to increase the reliability of vision-language models in deployment settings and highlights avenues for confidence calibration and prompt-design improvements.

Abstract

With the recent emergence of foundation models trained on internet-scale data and demonstrating remarkable generalization capabilities, such foundation models have become more widely adopted, leading to an expanding range of application domains. Despite this rapid proliferation, the trustworthiness of foundation models remains underexplored. Specifically, the out-of-distribution detection (OoDD) capabilities of large vision-language models (LVLMs), such as GPT-4o, which are trained on massive multi-modal data, have not been sufficiently addressed. The disparity between their demonstrated potential and practical reliability raises concerns regarding the safe and trustworthy deployment of foundation models. To address this gap, we evaluate and analyze the OoDD capabilities of various proprietary and open-source LVLMs. Our investigation contributes to a better understanding of how these foundation models represent confidence scores through their generated natural language responses. Furthermore, we propose a self-guided prompting approach, termed Reflexive Guidance (ReGuide), aimed at enhancing the OoDD capability of LVLMs by leveraging self-generated image-adaptive concept suggestions. Experimental results demonstrate that our ReGuide enhances the performance of current LVLMs in both image classification and OoDD tasks. The lists of sampled images, along with the prompts and responses for each sample are available at https://github.com/daintlab/ReGuide.

Figures (17)

• Figure 1: Performance of proprietary and open-source LVLMs on the ImageNet200 benchmark: (a) state-of-the-art comparison and (b) the boosting effect of our ReGuide
• Figure 2: Comparison of the OoDD framework for single-modal classifiers, CLIP, and LVLMs
• Figure 3: Framework for OoDD evaluation on LVLMs
• Figure 4: Further analysis on the ImageNet200 benchmark. (a) FPR across different TPR thresholds, (b) OoD score distribution, (c) OoD detectability based on model sizes, and (d) ID classification accuracy and OoDD performance of InternVL2-26B according to the given class order
• Figure 5: Framework of the proposed Reflexive Guidance for OoDD