Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalizable Vision-Language Few-Shot Adaptation with Predictive Prompts and Negative Learning

Sriram Mandalika

TL;DR

PromptFuseNL addresses the challenge of few-shot vision-language adaptation under limited supervision and noisy support by unifying predictive prompts, dual-branch prototype alignment, cross-modal coordination, and a lightweight robustness mechanism. The approach learns both what a class is and is not by combining positive prototype alignment with hard negative mining, while conditioning textual prompts and grounding them with visual context. It introduces a residual-enhanced, cross-modal prototype framework and an unsupervised instance reweighting strategy that downweights unreliable support samples, all without updating the frozen VLM backbone. Empirically, PromptFuseNL achieves state-of-the-art performance across 15 benchmarks, exhibits strong robustness to label noise and distribution shifts, and delivers substantial efficiency gains (e.g., up to 300x faster training and 1000x lower FLOPs) compared with full prompt tuning, making it highly practical for scalable few-shot VLM adaptation.

Abstract

Few-shot adaptation remains a core challenge for vision-language models (VLMs), especially under limited supervision and noisy support samples. We propose PromptFuseNL, a unified framework that enhances few-shot generalization by combining predictive prompt tuning with dual-branch positive and negative learning. The method refines class prototypes through task-conditioned residuals, multi-stage cross-modal coordination, and semantic hard negative mining. To address label noise, we introduce an unsupervised instance reweighting strategy that downweights unreliable support examples without requiring additional labels or structural changes. PromptFuseNL fuses visual and textual cues through lightweight modules for efficient and discriminative prediction. Evaluated across 15 benchmarks, it consistently surpasses existing prompt- and adapter-based methods in all shot settings while remaining highly efficient, achieving up to 300x faster training and 1000x lower FLOPs compared to full prompt tuning, achieving a new state-of-the-art for robust and scalable few-shot vision-language adaptation.

Generalizable Vision-Language Few-Shot Adaptation with Predictive Prompts and Negative Learning

TL;DR

PromptFuseNL addresses the challenge of few-shot vision-language adaptation under limited supervision and noisy support by unifying predictive prompts, dual-branch prototype alignment, cross-modal coordination, and a lightweight robustness mechanism. The approach learns both what a class is and is not by combining positive prototype alignment with hard negative mining, while conditioning textual prompts and grounding them with visual context. It introduces a residual-enhanced, cross-modal prototype framework and an unsupervised instance reweighting strategy that downweights unreliable support samples, all without updating the frozen VLM backbone. Empirically, PromptFuseNL achieves state-of-the-art performance across 15 benchmarks, exhibits strong robustness to label noise and distribution shifts, and delivers substantial efficiency gains (e.g., up to 300x faster training and 1000x lower FLOPs) compared with full prompt tuning, making it highly practical for scalable few-shot VLM adaptation.

Abstract

Few-shot adaptation remains a core challenge for vision-language models (VLMs), especially under limited supervision and noisy support samples. We propose PromptFuseNL, a unified framework that enhances few-shot generalization by combining predictive prompt tuning with dual-branch positive and negative learning. The method refines class prototypes through task-conditioned residuals, multi-stage cross-modal coordination, and semantic hard negative mining. To address label noise, we introduce an unsupervised instance reweighting strategy that downweights unreliable support examples without requiring additional labels or structural changes. PromptFuseNL fuses visual and textual cues through lightweight modules for efficient and discriminative prediction. Evaluated across 15 benchmarks, it consistently surpasses existing prompt- and adapter-based methods in all shot settings while remaining highly efficient, achieving up to 300x faster training and 1000x lower FLOPs compared to full prompt tuning, achieving a new state-of-the-art for robust and scalable few-shot vision-language adaptation.
Paper Structure (22 sections, 16 equations, 2 figures, 2 tables)

This paper contains 22 sections, 16 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Problem Formulation
  5. Core Learning Objectives
  6. Positive Prototype Alignment
  7. Negative Learning and Hard Negative Mining
  8. Contrast with Instance-Level Contrastive Learning
  9. Modality-Specific Branch Designs
  10. Textual Branch
  11. Prompt Prediction & Style Conditioning.
  12. Cross‑modal Coordination.
  13. Visual Branch
  14. Robustness Mechanism
  15. Instance Reweighting for Noise-Robust Prototype Refinement
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Illustration of the PromptFuseNL architecture, integrating dual-branch vision-language adaptation using learnable prompt prediction, residual-enhanced prototypes, and cross-modal fusion. Both textual and visual features are aligned and contrasted with query features through cosine similarity.
  • Figure 2: Performance comparisons on few-shot learning on 11 image classification datasets. For each dataset, we report the mean accuracy over 3 random seeds of our PromptFuseNL on 1-/2-/4-/8-/16-shot settings.