XR-VLM: Cross-Relationship Modeling with Multi-part Prompts and Visual Features for Fine-Grained Recognition

TL;DR

The paper tackles fine-grained visual recognition (FGVR) by addressing the limitations of alignment-based predictions in vision-language models (VLMs). It introduces XR-VLM, a framework that combines multi-part prompts and multi-part visual features through a Unified Attention module and a cross-relationship modeling (CRM) scheme, enabling rich cross-modal interactions via a cross-relationship representation $R = \epsilon(V \otimes T)$ and a classifier $\hat{y} = \vartheta(R)$. Empirical results on five FGVR benchmarks show significant gains over state-of-the-art methods for both RN50 and ViT backbones, with ablations confirming the importance of cross-relations and multi-part learning. The approach offers a scalable, efficient path to better FGVR performance, with code to be released, and lays groundwork for extending cross-modal relationships in VLM adaptations.

Abstract

Vision-Language Models (VLMs) have demonstrated impressive performance on various visual tasks, yet they still require adaptation on downstream tasks to achieve optimal performance. Recently, various adaptation technologies have been proposed, but we observe they often underperform in fine-grained visual recognition, which requires models to capture subtle yet discriminative features to distinguish similar sub-categories. Current adaptation methods typically rely on an alignment-based prediction framework, \ie the visual feature is compared with each class prompt for similarity calculation as the final prediction, which lacks class interaction during the forward pass. Besides, learning single uni-modal feature further restricts the model's expressive capacity. Therefore, we propose a novel mechanism, XR-VLM, to discover subtle differences by modeling cross-relationships, which specifically excels in scenarios involving multiple features. Our method introduces a unified multi-part visual feature extraction module designed to seamlessly integrate with the diverse backbones inherent in VLMs. Additionally, we develop a multi-part prompt learning module to capture multi-perspective descriptions of sub-categories. To further enhance discriminative capability, we propose a cross relationship modeling pattern that combines visual feature with all class prompt features, enabling a deeper exploration of the relationships between these two modalities. Extensive experiments have been conducted on various fine-grained datasets, and the results demonstrate that our method achieves significant improvements compared to current state-of-the-art approaches. Code will be released.

Figures (12)

• Figure 1: Comparison between (a) previous prediction pattern, generated by aligning a single visual feature with each class prompt feature individually (referred to as the aligning pattern), and (b) our prediction pattern, generated by modeling cross relationship between a single visual feature and all class prompt features collectively (referred to as the crossing pattern)
• Figure 2: Overall framework of our proposed XR-VLM: For the text branch, multi-part learnable prompts of classes are fed into the Text Encoder, generating multi-part prompt features. For the image branch, the input image is processed through the Image Encoder and an Unified Attention module to generate multi-part visual features. These prompt and visual features are sent to generated the cross-relationship representations, which is finally sent to a MLP module to generate the predictions. (best view in color)
• Figure 3: Illustration of cross relationships. Different shape borders represent different parts. (best view in color)
• Figure 4: Illustration of different cross relationships for prediction.
• Figure 5: Comparison between PLOT and our method with different number of Prompt/Visual features and image encoders.