Open-Set Recognition in the Age of Vision-Language Models

TL;DR

Vision-language models are not intrinsically open-set; their finite query vocabulary imposes a closed-set bias that becomes problematic under open-set conditions. The authors redefine the open-set problem for VLMs, propose a standardized benchmark and evaluation protocol, and systematically assess baseline strategies based on predictive uncertainty and negative embeddings across multiple VLMs and open-vocabulary detectors. They find that state-of-the-art VLMs struggle with open-set recognition, that enlarging the query set is ineffective and can hurt task performance, and that negative embeddings offer partial mitigation with notable trade-offs. The work provides a unified evaluation framework and actionable insights into open-set vulnerabilities, informing safer deployment and guiding future research in robust open-set recognition for VLMs.

Abstract

Are vision-language models (VLMs) for open-vocabulary perception inherently open-set models because they are trained on internet-scale datasets? We answer this question with a clear no - VLMs introduce closed-set assumptions via their finite query set, making them vulnerable to open-set conditions. We systematically evaluate VLMs for open-set recognition and find they frequently misclassify objects not contained in their query set, leading to alarmingly low precision when tuned for high recall and vice versa. We show that naively increasing the size of the query set to contain more and more classes does not mitigate this problem, but instead causes diminishing task performance and open-set performance. We establish a revised definition of the open-set problem for the age of VLMs, define a new benchmark and evaluation protocol to facilitate standardised evaluation and research in this important area, and evaluate promising baseline approaches based on predictive uncertainty and dedicated negative embeddings on a range of open-vocabulary VLM classifiers and object detectors.

Figures (12)

• Figure 1: Left: Traditional open-set recognition arises when testing classes outside a model's finite training class set. While VLMs are trained on internet-scale datasets containing most conceivable object classes, they use a finite query set for classification. Open-set recognition arises when test classes are not included in the query set. Right: When testing an object present in the predefined query set, VLMs often correctly classify the object. When testing an object that is not present in the query set (i.e. an open-set object), VLMs often misclassify the object as a query class with high confidence (i.e. an open-set error).
• Figure 2: The trade-off between closed-set and open-set performance with increasing number of negative queries for different VLM classifiers.
• Figure 3: The trade-off between closed-set and open-set performance with increasing number of negative queries for different VLM object detectors.
• Figure 4: The relationship between closed-set and open-set performance in VLMs.
• Figure 5: Influence of the query set size on closed-set and open-set performance on ImageNet1k classification, showing mean and standard deviation from 10 random seeds.