Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models

TL;DR

This work tackles zero-shot fine-grained image classification by reframing the task as visual question answering with Large Vision-Language Models (LVLMs). It introduces an iterative multiple-choice question-answering (MCQA) pipeline and a lightweight attention intervention to better align early visual cues with deep semantic grounding, plus curated, attribute-rich class descriptions to bridge the gap between visuals and discriminative attributes. The authors demonstrate substantial, consistent improvements over prior zero-shot methods across five fine-grained benchmarks, achieving state-of-the-art performance in many settings, and analyze trade-offs between iterative MCQA and all-at-once inference, model scale, and description quality. The results highlight the practical potential of LVLM-driven zero-shot fine-grained classification, especially when combined with precise descriptions and targeted attention guidance, with implications for scalable, training-free recognition in visually similar categories.

Abstract

Large Vision-Language Models (LVLMs) have demonstrated impressive performance on vision-language reasoning tasks. However, their potential for zero-shot fine-grained image classification, a challenging task requiring precise differentiation between visually similar categories, remains underexplored. We present a novel method that transforms zero-shot fine-grained image classification into a visual question-answering framework, leveraging LVLMs' comprehensive understanding capabilities rather than relying on direct class name generation. We enhance model performance through a novel attention intervention technique. We also address a key limitation in existing datasets by developing more comprehensive and precise class description benchmarks. We validate the effectiveness of our method through extensive experimentation across multiple fine-grained image classification benchmarks. Our proposed method consistently outperforms the current state-of-the-art (SOTA) approach, demonstrating both the effectiveness of our method and the broader potential of LVLMs for zero-shot fine-grained classification tasks. Code and Datasets: https://github.com/Atabuzzaman/Fine-grained-classification

Figures (6)

• Figure 1: Overview of our zero-shot fine-grained image classification framework. Unlike existing approaches (top), which directly prompt a Large Vision-Language Model (LVLM) to generate a class name given an input image, our method leverages an LVLM combined with a proposed iterative multiple-choice question-answering strategy and an attention intervention technique to select the most accurate fine-grained class description. This framework effectively matches each input image with the most appropriate class description without requiring any training samples.
• Figure 2: Overview of our proposed zero-shot fine-grained image classification framework using LVLM. The system takes an input image and class descriptions with a prompt, and uses an LVLM enhanced with an attention intervention mechanism. The framework employs an iterative MCQA approach where the LVLM selects the most appropriate class description through multiple rounds of refinement. The attention intervention module guides the visual information flow from shallow to deep layers, while deep layers provide grounded object-attribute information to final layers to improve classification accuracy.
• Figure 3: Comparison of class descriptions from an existing dataset and our introduced class descriptions. Bold text in the "Ours Class Descriptions" column highlights key discriminative features that are either absent or described with less specificity in the "Existing Class Descriptions" column. The increased detail in our proposed descriptions facilitates more accurate zero-shot fine-grained image classification.
• Figure 4: Prompt used for the iterative multiple-choice question answering (MCQA) approach.
• Figure 5: Prompt used for obtaining fine-grained visual descriptions from an LVLM.