Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models

TL;DR

The paper addresses the challenge of zero-/few-shot fine-grained image classification, where CLIP-based vision-language models struggle to distinguish semantically similar sub-classes. It introduces CascadeVLM, a cascaded framework that first uses CLIP to filter candidate classes and then leverages large vision-language models (LVLMs) with zero-shot or few-shot prompts to refine predictions, guided by an adaptive entropy threshold for efficiency. Across six fine-grained datasets, CascadeVLM achieves superior zero-shot performance and notable gains in few-shot settings, with key insights into candidate ordering, efficiency trade-offs, and explainable reasoning. The approach demonstrates a practical pathway to integrate CLIP-like backbones with LVLMs for accurate, efficient, and interpretable fine-grained classification in real-world scenarios.

Abstract

Fine-grained image classification, particularly in zero/few-shot scenarios, presents a significant challenge for vision-language models (VLMs), such as CLIP. These models often struggle with the nuanced task of distinguishing between semantically similar classes due to limitations in their pre-trained recipe, which lacks supervision signals for fine-grained categorization. This paper introduces CascadeVLM, an innovative framework that overcomes the constraints of previous CLIP-based methods by effectively leveraging the granular knowledge encapsulated within large vision-language models (LVLMs). Experiments across various fine-grained image datasets demonstrate that CascadeVLM significantly outperforms existing models, specifically on the Stanford Cars dataset, achieving an impressive 85.6% zero-shot accuracy. Performance gain analysis validates that LVLMs produce more accurate predictions for challenging images that CLIPs are uncertain about, bringing the overall accuracy boost. Our framework sheds light on a holistic integration of VLMs and LVLMs for effective and efficient fine-grained image classification.

Figures (10)

• Figure 1: Illustration of model performance: CLIP's misclassification of watercress (left) and the inverse relationship between LVLM accuracy and the number of categories (right).
• Figure 2: CascadeVLM commences with CLIP for initial image analysis and probabilistic categorization, integrating an entropy threshold, $\tau$, to balance efficiency and accuracy, culminating in LVLM's adaptive classification.
• Figure 3: Comparative Analysis of ACC performance between CLIP and GPT-4V across different intervals of classification certainty. The left graph shows the ACC of both models across varying levels of margin. The right graph presents the ACC gap between the two models.
• Figure 4: Performance variation in the StanfordCars dataset with varying entropy thresholds using CLIP-ViT-L/14 for cascading, set at top-k=10. An increase in entropy threshold results in decreased inference speed and reduced accuracy.
• Figure 5: CLIP's prediction is incorrect, while LVLM corrects the answer and makes the reasoning.