Cross-modal Proxy Evolving for OOD Detection with Vision-Language Models

TL;DR

CoEvo introduces a test-time, bidirectional cross-modal framework for zero-shot OOD detection that evolves textual negatives and visual proxies in a coordinated loop. By maintaining separate textual and visual proxy caches and coupling their updates through a proxy-aligned co-evolution mechanism, CoEvo realigns cross-modal similarities under distribution shift without updating the backbone. A dynamic multi-modal score, together with adaptive thresholding and confidence-based updates, yields calibrated OOD decisions and improved robustness. Experiments on ImageNet-1K and OpenOOD demonstrate state-of-the-art performance, with notable reductions in FPR95 and gains in AUROC compared to strong negative-label baselines. The approach highlights the value of online, cross-modal adaptation for open-world recognition and offers a practical, training-free solution for robust vision–language OOD detection.

Abstract

Reliable zero-shot detection of out-of-distribution (OOD) inputs is critical for deploying vision-language models in open-world settings. However, the lack of labeled negatives in zero-shot OOD detection necessitates proxy signals that remain effective under distribution shift. Existing negative-label methods rely on a fixed set of textual proxies, which (i) sparsely sample the semantic space beyond in-distribution (ID) classes and (ii) remain static while only visual features drift, leading to cross-modal misalignment and unstable predictions. In this paper, we propose CoEvo, a training- and annotation-free test-time framework that performs bidirectional, sample-conditioned adaptation of both textual and visual proxies. Specifically, CoEvo introduces a proxy-aligned co-evolution mechanism to maintain two evolving proxy caches, which dynamically mines contextual textual negatives guided by test images and iteratively refines visual proxies, progressively realigning cross-modal similarities and enlarging local OOD margins. Finally, we dynamically re-weight the contributions of dual-modal proxies to obtain a calibrated OOD score that is robust to distribution shift. Extensive experiments on standard benchmarks demonstrate that CoEvo achieves state-of-the-art performance, improving AUROC by 1.33% and reducing FPR95 by 45.98% on ImageNet-1K compared to strong negative-label baselines.

Figures (7)

• Figure 1: Proxy-aligned co-evolution. For each test sample, textual negatives are dynamically mined to expand the occupied space around its semantic context, while visual positive/negative proxies are updated online. This joint evolution maintains aligned cross-modal similarities under distribution shift, enabling robust zero-shot OOD decisions.
• Figure 2: Pipeline of the proposed cross-modal proxy co-evolving framework (CoEvo). It dynamically updates both visual and textual proxy caches based on high-confidence samples, enabling bidirectional alignment and robust zero-shot OOD detection.
• Figure 3: Sensitivity to the fusion weight $\lambda$. Results are reported on the ImageNet-1K benchmark.
• Figure 4: Sensitivity analysis of the hyperparameter $\operatorname{Top}\text{-}N$, evaluated on the ImageNet-1K benchmark.
• Figure 5: Analyses on the hyper-parameter of queue length $L$, where results are reported on ImageNet-1K benchmark with CoEvo$_\mathrm{CSP}$.