OVMR: Open-Vocabulary Recognition with Multi-Modal References

TL;DR

OVMR tackles open-vocabulary recognition by combining textual descriptions with exemplar images to form robust, multi-modal category cues. It introduces a lightweight visual token generator to produce visual tokens from exemplars and fuses them with language cues through a two-stage process: multi-modal classifier generation and a test-time preference-based fusion that requires no trainable fusion parameters. The method is shown to outperform prompting and several few-shot baselines on 11 classification datasets and LVIS in detection, while remaining robust to exemplar quality and enabling plug-and-play deployment with internet-sourced exemplars. The practical impact lies in efficient, scalable open-vocabulary recognition that leverages diverse, readily available cross-modal clues without heavy retraining. OV MR’s adaptable framework promises improved generalization for real-world open-set recognition tasks and can be extended to other multi-modal downstream tasks.

Abstract

The challenge of open-vocabulary recognition lies in the model has no clue of new categories it is applied to. Existing works have proposed different methods to embed category cues into the model, \eg, through few-shot fine-tuning, providing category names or textual descriptions to Vision-Language Models. Fine-tuning is time-consuming and degrades the generalization capability. Textual descriptions could be ambiguous and fail to depict visual details. This paper tackles open-vocabulary recognition from a different perspective by referring to multi-modal clues composed of textual descriptions and exemplar images. Our method, named OVMR, adopts two innovative components to pursue a more robust category cues embedding. A multi-modal classifier is first generated by dynamically complementing textual descriptions with image exemplars. A preference-based refinement module is hence applied to fuse uni-modal and multi-modal classifiers, with the aim to alleviate issues of low-quality exemplar images or textual descriptions. The proposed OVMR is a plug-and-play module, and works well with exemplar images randomly crawled from the Internet. Extensive experiments have demonstrated the promising performance of OVMR, \eg, it outperforms existing methods across various scenarios and setups. Codes are publicly available at \href{https://github.com/Zehong-Ma/OVMR}{https://github.com/Zehong-Ma/OVMR}.

Figures (7)

• Figure 1: Illustration of the pipeline of our OVMR. It refers to textual description and exemplar images to generate classifiers for novel categories. The textual description could be ambiguous and fail to depict visual details. The exemplar images show diversified qualities. OVMR effectively complements visual and textual features and fuses classifiers to alleviate issues of low-quality exemplar images or textual description.
• Figure 2: Illustration of the pipeline for novel classifier generation (left) and image classification (right).
• Figure 3: The accuracy of various classifiers at varying shots.
• Figure 4: The variation in preference weight for different classifiers corresponding to multi-modal references of various qualities. (a) The category name "Chain" and the exemplar images are both of high quality and effectively complement each other. (b) The category name "Balloon Flower" may not describe the fine-grained flower in detail and is of low quality, whereas the exemplar images accurately represent the category. (c) The exemplar images with various backgrounds, poses, and appearances are of low quality, but the common word "Leopard" clearly defines the animal.
• Figure 5: Exemplar images sampled from the training set of ImageNet and the Internet-crawled images.