PMCE: Probabilistic Multi-Granularity Semantics with Caption-Guided Enhancement for Few-Shot Learning

TL;DR

PMCE addresses the bias in prototypes in 1-shot learning by leveraging a nonparametric knowledge bank of base-class statistics and CLIP-based class-name semantics to form class-specific priors via MAP calibration. It further refines representations through a caption-guided enhancer that uses label-free BLIP captions to update both support prototypes and query features in the same metric space, all while keeping backbones frozen. The approach yields consistent gains across four benchmarks and two backbones, with the largest improvements in 1-shot tasks, demonstrating that semantic priors plus instance-level captions can substantially stabilize and align representations without heavy model fine-tuning. This framework has practical impact for scalable, data-efficient recognition in settings where labeled data are scarce and rapid adaptation is required, and it lays groundwork for integrating caption reliability into few-shot learning pipelines.

Abstract

Few-shot learning aims to identify novel categories from only a handful of labeled samples, where prototypes estimated from scarce data are often biased and generalize poorly. Semantic-based methods alleviate this by introducing coarse class-level information, but they are mostly applied on the support side, leaving query representations unchanged. In this paper, we present PMCE, a Probabilistic few-shot framework that leverages Multi-granularity semantics with Caption-guided Enhancement. PMCE constructs a nonparametric knowledge bank that stores visual statistics for each category as well as CLIP-encoded class name embeddings of the base classes. At meta-test time, the most relevant base classes are retrieved based on the similarities of class name embeddings for each novel category. These statistics are then aggregated into category-specific prior information and fused with the support set prototypes via a simple MAP update. Simultaneously, a frozen BLIP captioner provides label-free instance-level image descriptions, and a lightweight enhancer trained on base classes optimizes both support prototypes and query features under an inductive protocol with a consistency regularization to stabilize noisy captions. Experiments on four benchmarks show that PMCE consistently improves over strong baselines, achieving up to 7.71% absolute gain over the strongest semantic competitor on MiniImageNet in the 1-shot setting. Our code is available at https://anonymous.4open.science/r/PMCE-275D

Figures (8)

• Figure 1: Comparison of semantic cues for support and query images. (a) Class-level semantics only (baseline). The support feature is modulated by its class name "robin", while the query remains purely visual, leading to a relatively weak alignment between them (cosine similarity 0.56). (b) Caption-guided support and query (PMCE). BLIP-generated captions provide label-free instance-level semantics for both images, yielding more focused activation maps and a stronger support–query alignment (cosine similarity 0.90).
• Figure 2: Overall framework of PMCE. Training stage: On base classes, we build a non-parametric knowledge bank that stores class-wise visual statistics and class-name embeddings, and train a caption-guided enhancer $\Phi$ by fusing visual features from $f$ with BLIP-generated caption semantics from $g$. Test stage: (1) extract visual features and label-free captions for both supports and queries; (2) use class-name semantics to retrieve related base classes from the knowledge bank and obtain a class-specific prior, which is fused with the support prototype in a MAP-style update to get $\tilde{p}$; (3) apply $\Phi$ to enhance $\tilde{p}$ with aggregated support captions and to enhance query features with their own captions, and perform metric-based classification in the aligned feature space.
• Figure 3: Sensitivity of PMCE to the prior weight $\alpha$ and the number of retrieved base classes $k$ on MiniImageNet. Performance is stable over a wide range of values, and our default settings lie in the high-accuracy region.
• Figure 4: Grad-CAM visualizations on the MiniImageNet test set. Compared with w/o caption, w/ caption produces more focused activations on query images, suppressing background responses and highlighting discriminative object regions.
• Figure 5: t-SNE visualization of feature distributions and prototype evolution on a 5-way 1-shot task from MiniImageNet. Different colors denote classes; $\star$ are ground-truth centers, $\blacksquare$ initial prototypes, $\triangle$ MAP rectified prototypes,large $\bullet$ caption-enhanced prototypes, and $\diamond$ the final prototypes of PMCE. MAP calibration and caption-guided enhancement jointly move prototypes toward the true centers and make clusters more compact.