Learning Optimal Prompt Ensemble for Multi-source Visual Prompt Transfer

TL;DR

We address efficient adaptation of large vision transformers by learning an optimal ensemble over multiple source prompts. HGPrompt introduces a differentiable transferability metric to quantify the informativeness of prompt-induced features and a gradient alignment regularization to suppress cross-prompt interference, optimizing prompt weights under a convex objective. The approach jointly tunes the target prompt and the header, enabling robust, scalable transfer across diverse VTAB tasks. Empirical results on VTAB with ViT-B/16 show state-of-the-art average accuracy and strong performance on fine-grained and reasoning tasks, validating the effectiveness of dynamic multi-source prompt transfer. This framework offers interpretability of source-task relevance and a practical pathway for scalable, privacy-preserving model adaptation.

Abstract

Prompt tuning has emerged as a lightweight strategy for adapting foundation models to downstream tasks, particularly for resource-constrained systems. As pre-trained prompts become valuable assets, combining multiple source prompts offers a promising approach to enhance generalization for new tasks by leveraging complementary knowledge. However, naive aggregation often overlooks different source prompts have different contribution potential to the target task. To address this, we propose HGPrompt, a dynamic framework that learns optimal ensemble weights. These weights are optimized by jointly maximizing an information-theoretic metric for transferability and minimizing gradient conflicts via a novel regularization strategy. Specifically, we propose a differentiable prompt transferability metric to captures the discriminability of prompt-induced features on the target task. Meanwhile, HGPrompt match the gradient variances with respect to different source prompts based on Hessian and Fisher Information, ensuring stable and coherent knowledge transfer while suppressing gradient conflicts among them. Extensive experiments on the large-scale VTAB benchmark demonstrate the state-of-the-art performance of HGPrompt, validating its effectiveness in learning an optimal ensemble for effective multi-source prompt transfer.

Figures (6)

• Figure 1: Multi-source prompt transfer framework. Task-specific prompts are tuned via a frozen backbone and statically aggregated for target initialization. Our approach dynamically optimizes source weights through single forward-backward propagation, learning prompt aggregation via an optimization module.
• Figure 2: Overview of the framework. Given an input image $X$, the system generates $M$ distinct feature representations $\{f_i\}_{i=1}^M$ and corresponding gradients $\{g^i\}_{i=1}^M$ through multiple source prompts. These features and gradients are fused using learnable weights $\boldsymbol{\alpha}$ to produce the final combined feature $f_{\boldsymbol{\alpha}}$ and gradient $g^{\boldsymbol{\alpha}}$. The Transferability term evaluates the fused feature distribution against the target class label, and the Gradient Alignment Regularization aligns the prompt gradient variance. The Weight Optimizer jointly optimizes these dual objectives to determine the optimal source weights $\boldsymbol{\alpha}$, which subsequently initialize the target prompt.
• Figure 3: Loss landscapes for a two-parameter model, showing conflicting gradient variances $\{g_i^{(A)}\}_{i=1}^{n_A}$ and $\{g_i^{(B)}\}_{i=1}^{n_B}$ around $\theta^*$. This shows the case where a nearby solution $\theta \in N_{A,\theta^*}^{\epsilon}$ maintains equivalent risk $\mathcal{R}_A(\theta) \approx \mathcal{R}_A(\theta^*)$ in domain A but exhibits higher risk in domain B.
• Figure 4: Prompt Weights analysis for 12 source prompts. Bar plots represent single-source transfer accuracy (left axis), while line plots indicate prompt weights (right axis).
• Figure 5: Performance scaling with increasing source prompts on Clever-Count target domain.

Theorems & Definitions (5)

• Definition 1
• Definition 2
• Theorem 1
• Definition 3
• Theorem 2