microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classification

TL;DR

This work tackles unsupervised adaptation of CLIP for fine-grained image classification, where global CLS representations miss localized discriminative cues. It introduces SOAP within a TokenFusion module to create a fine-grained FG token from patch embeddings via a saliency-guided NCut, and fuses FG with CLS to achieve coarse–fine alignment. A two-headed LLM-derived classifier provides both a stable prior and a learnable component, and Dynamic Knowledge Aggregation convexly blends static priors with evolving TokenFusion logits to refine pseudo-labels. Empirically, microCLIP delivers a 2.90 percentage-point improvement on 13 fine-grained benchmarks with lightweight adaptation, revealing latent fine-grained signals in CLIP and offering a practical pathway for unsupervised fine-grained CLIP adaptation.

Abstract

Unsupervised adaptation of CLIP-based vision-language models (VLMs) for fine-grained image classification requires sensitivity to microscopic local cues. While CLIP exhibits strong zero-shot transfer, its reliance on coarse global features restricts its performance on fine-grained classification tasks. Prior efforts inject fine-grained knowledge by aligning large language model (LLM) descriptions with the CLIP $\texttt{[CLS]}$ token; however, this approach overlooks spatial precision. We propose $\textbf{microCLIP}$, a self-training framework that jointly refines CLIP's visual and textual representations using fine-grained cues. At its core is Saliency-Oriented Attention Pooling (SOAP) within a lightweight TokenFusion module, which builds a saliency-guided $\texttt{[FG]}$ token from patch embeddings and fuses it with the global $\texttt{[CLS]}$ token for coarse-fine alignment. To stabilize adaptation, we introduce a two-headed LLM-derived classifier: a frozen classifier that, via multi-view alignment, provides a stable text-based prior for pseudo-labeling, and a learnable classifier initialized from LLM descriptions and fine-tuned with TokenFusion. We further develop Dynamic Knowledge Aggregation, which convexly combines fixed LLM/CLIP priors with TokenFusion's evolving logits to iteratively refine pseudo-labels. Together, these components uncover latent fine-grained signals in CLIP, yielding a consistent $2.90\%$ average accuracy gain across 13 fine-grained benchmarks while requiring only light adaptation. Our code is available at https://github.com/sathiiii/microCLIP.

Figures (11)

• Figure 1: Attention maps on two fine-grained datasets: Birdsnap and RESISC45. Row (a): input images; (b): global attention from DPA DPA; (c): local attention from microCLIP (ours). By guiding the [FG] token with SOAP queries, microCLIP focuses on semantically critical regions, yielding sharper, more discriminative attention. Red circles highlight referenced regions in the text.
• Figure 2: Overall architecture of microCLIP. The top shows our pseudo-labeling pipeline, where fixed knowledge from CLIP via the alignment between multi-view augmented representations and fine-grained LLM-generated descriptions is combined with dynamic knowledge learned in TokenFusion. The bottom illustrates our TokenFusion module.
• Figure 3: Pseudo-Labeling Accuracy variation of each component and Dynamic Knowledge Aggregation over time on the Stanford Cars train split.
• Figure 4: NCut-based saliency masks on bird images from Birdsnap birdsnap. Top: input images; bottom: salient regions after CRF refinement.
• Figure 5: $\gamma$ sensitivity analysis on the DTD dataset.