LLM meets Vision-Language Models for Zero-Shot One-Class Classification

TL;DR

Zero-shot one-class classification identifies whether a query image belongs to a target class using only the class label. The proposed approach combines LLM-generated visually confusing negatives with CLIP-style cosine similarity in a per-task adaptive boundary, formalized via $s(\\mathbf{x},\\mathbf{t})$ and a threshold $\\lambda_{\\mathbf{t}}(\\mathbf{x})$, with a hybrid ANP+FT scheme that blends adaptive and fixed thresholds. A realistic iNaturalist-based benchmark with hierarchical and uniform task sampling demonstrates state-of-the-art macro F1 across fine- and coarse-grained tasks, while ablations confirm the superiority of average negative prototypes and the value of combining adaptive and fixed thresholds. The work offers practical flexibility for dynamic target concepts and highlightsthreshold calibration as a key factor in zero-shot one-class discrimination, though it acknowledges dependencies on LLM prompts and the transferability of ImageNet-derived thresholds as areas for future improvement.

Abstract

We consider the problem of zero-shot one-class visual classification, extending traditional one-class classification to scenarios where only the label of the target class is available. This method aims to discriminate between positive and negative query samples without requiring examples from the target class. We propose a two-step solution that first queries large language models for visually confusing objects and then relies on vision-language pre-trained models (e.g., CLIP) to perform classification. By adapting large-scale vision benchmarks, we demonstrate the ability of the proposed method to outperform adapted off-the-shelf alternatives in this setting. Namely, we propose a realistic benchmark where negative query samples are drawn from the same original dataset as positive ones, including a granularity-controlled version of iNaturalist, where negative samples are at a fixed distance in the taxonomy tree from the positive ones. To our knowledge, we are the first to demonstrate the ability to discriminate a single category from other semantically related ones using only its label.

Figures (13)

• Figure 1: In the zero-shot one-class classification problem, the class is defined using a text prompt and the task is to accept or reject query images independently. As the prompt text varies, the decision boundaries of the class also varies. Such a task is challenging as it requires an accurate estimate of the boundaries of each class.
• Figure 2: Illustration of the proposed method for two instances of a zero-shot one-class classification task, where only the class label is given. In the first case (a), only the "Finch" example is identified as a "Finch". In the second scenario (b), both bird examples are classified as birds. Negative examples of classes are obtained through querying an LLM. Notice that the position of query images is unchanged between the two instances, as only the class boundaries vary.
• Figure 3: Distribution of the optimal threshold for each task for different levels in iNaturalist.
• Figure 3: Ablation study showing the average macro F1 scores for different modules.
• Figure 4: Macro F1 score per level in iNaturalist for different methods.