Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMs

TL;DR

The paper addresses vocabulary-free fine-grained visual recognition by leveraging weak labels from Multimodal LLMs and refining them with a nearest-neighbor guided mechanism. The proposed NeaR framework constructs candidate label sets, partitions data into clean and noisy samples with a Gaussian Mixture Model, refines labels using candidate information, and filters labels at inference to keep the output space compact. Empirical results across five FGVR datasets show NeaR outperforms direct MLLM inference and several VF baselines while substantially reducing computation time and cost, including strong performance with open-source MLLMs. The work demonstrates that open-ended MLLM outputs can be harnessed effectively for scalable VF-FGVR, enabling robust fine-grained recognition in data-constrained domains.

Abstract

Fine-grained Visual Recognition (FGVR) involves distinguishing between visually similar categories, which is inherently challenging due to subtle inter-class differences and the need for large, expert-annotated datasets. In domains like medical imaging, such curated datasets are unavailable due to issues like privacy concerns and high annotation costs. In such scenarios lacking labeled data, an FGVR model cannot rely on a predefined set of training labels, and hence has an unconstrained output space for predictions. We refer to this task as Vocabulary-Free FGVR (VF-FGVR), where a model must predict labels from an unconstrained output space without prior label information. While recent Multimodal Large Language Models (MLLMs) show potential for VF-FGVR, querying these models for each test input is impractical because of high costs and prohibitive inference times. To address these limitations, we introduce \textbf{Nea}rest-Neighbor Label \textbf{R}efinement (NeaR), a novel approach that fine-tunes a downstream CLIP model using labels generated by an MLLM. Our approach constructs a weakly supervised dataset from a small, unlabeled training set, leveraging MLLMs for label generation. NeaR is designed to handle the noise, stochasticity, and open-endedness inherent in labels generated by MLLMs, and establishes a new benchmark for efficient VF-FGVR.

Figures (5)

• Figure 1: Overview of our proposed method, NeaR, for Vocabulary-Free Fine-Grained Visual Recognition (VF-FGVR). In the Training Stage, we start with a few unlabeled images. Step-1: An MLLM generates a best-estimate fine-grained label (e.g., "Red-headed woodpecker") for each image. Step-2: A candidate label set is constructed using K-Nearest Neighbors, capturing related fine-grained classes. Step-3: The model is fine-tuned using a CLIP-based architecture. A GMM is applied to the loss to partition the data into clean and noisy samples. Based on this split, a label refinement mechanism is used to further update and refine the labels. The final loss, $L_{final}$, is then computed, and the model parameters are updated accordingly. In the Inference Stage, we apply label filtering to limit the label space (Step-4). Our approach handles the noise and open-ended nature of MLLM-generated labels, significantly reducing inference time and cost while maintaining performance.
• Figure 2: Effect of varying $m$, number of images per class in training data for labels generated by LLaMA. NeaR (in purple) outperforms CoOp (blue) for all $m\geq 2$ in average cACC & sACC.
• Figure A3: Effect of varying $\kappa$ (1 to 9) on the performance of NeaR-LLaMA for the 9-shot Flowers-102 dataset. The results show an upward trend in cACC as $\kappa$ increases from 1 to 3, reflecting an increased likelihood of semantically closer labels. However, for $\kappa \geq 3$, the performance plateaus or slightly decreases due to a noisier candidate set, validating our choice of $\kappa=3$.
• Figure A4: We report cACC and sACC under the effect of random sampling of training images across five datasets. The plot demonstrates minimal variance across datasets, highlighting the robustness of NeaR to variations in data selection.
• Figure A5: We present qualitative results across five benchmark datasets, comparing predictions from LLaMA, ZS-CLIP-LLaMA, CoOp-LLaMA, and NeaR.