Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models

TL;DR

The paper addresses the FGVR weakness of multimodal large language models by diagnosing misalignment between visual objects and subordinate-level category names as the key bottleneck. It introduces Finedefics, which leverages per-sample attribute descriptions constructed via LLMs and VQA to create attribute-augmented alignment between objects, attributes, and categories, followed by a two-stage training regime that combines contrastive learning with classification-centered instruction tuning. Across six FGVR datasets, Finedefics yields substantial improvements over strong baselines, demonstrating the effectiveness of attribute-driven alignment in enhancing fine-grained recognition and informing downstream tasks. The work contributes a principled framework for integrating attribute semantics into FGVR, with broad potential to generalize to other MLLMs and to support continual FGVR learning in real-world applications.

Abstract

Multi-modal large language models (MLLMs) have shown remarkable abilities in various visual understanding tasks. However, MLLMs still struggle with fine-grained visual recognition (FGVR), which aims to identify subordinate-level categories from images. This can negatively impact more advanced capabilities of MLLMs, such as object-centric visual question answering and reasoning. In our study, we revisit three quintessential capabilities of MLLMs for FGVR, including object information extraction, category knowledge reserve, object-category alignment, and position of the root cause as a misalignment problem. To address this issue, we present Finedefics, an MLLM that enhances the model's FGVR capability by incorporating informative attribute descriptions of objects into the training phase. We employ contrastive learning on object-attribute pairs and attribute-category pairs simultaneously and use examples from similar but incorrect categories as hard negatives, naturally bringing representations of visual objects and category names closer. Extensive evaluations across multiple popular FGVR datasets demonstrate that Finedefics outperforms existing MLLMs of comparable parameter sizes, showcasing its remarkable efficacy. The code is available at https://github.com/PKU-ICST-MIPL/Finedefics_ICLR2025.

Figures (7)

• Figure 1: Three quintessential capabilities of MLLMs for fine-grained visual recognition. Current MLLMs possess acceptable capabilities in image information extraction and category knowledge reserve but struggle with aligning objects to their corresponding subordinate-level categories.
• Figure 2: Object/Category/Object-category representation visualization of SigLIP and Idefics2.
• Figure 3: An illustration of framework to build Finedefics. (a) Attribute Description Construction, which aims to obtain informative attribute descriptions of objects. (b) Attribute Augmented Alignment, which aims to use constructed attribute descriptions to bind visual objects and category names, thus enhancing the model's FGVR capability via a two-stage training paradigm.
• Figure 4: Representation visualization of Finetune, CL (object-category) and Finedefics.
• Figure 5: Confusion matrix of Oxford-IIIT Pet-37.