Prompt Generation Networks for Input-Space Adaptation of Frozen Vision Transformers

TL;DR

Prompt Generation Networks enable input-space adaptation of frozen vision transformers by generating per-image prompts from a lightweight token library. This decouples adaptation from model internals, allowing client-side deployment and API-friendly usage while preserving strong performance across 12 datasets, often matching or surpassing full finetuning with two orders of magnitude fewer adapted parameters. The approach demonstrates clear benefits over fixed-domain prompts, scales to multiple datasets, and supports efficient multi-dataset training. The work highlights a practical path to leveraging large foundation models in constrained environments and provides insights into the division of labor between frozen encoders and learnable prompts.

Abstract

With the introduction of the transformer architecture in computer vision, increasing model scale has been demonstrated as a clear path to achieving performance and robustness gains. However, with model parameter counts reaching the billions, classical finetuning approaches are becoming increasingly limiting and even unfeasible when models become hosted as inference APIs, as in NLP. Visual input-prompt learning, an adaptation technique in which additional inputs in visual (RGB) space are learned, has emerged as a potential solution for adapting frozen and cloud-hosted models, requiring neither access to the forward pass, nor post-processing. Yet so far, these constraints have deteriorated adaptation performances significantly. To this end, we propose the Prompt Generation Network (PGN) that generates a different prompt for every data point, which is then used to adapt a frozen pretrained vision model to a target task. We show that the PGN effectively adapts pretrained models to various new datasets: It surpasses previous methods by a large margin on 12/12 datasets and even outperforms full-finetuning on 5/12, while requiring 100x fewer parameters. Lastly, we introduce the "prompt inversion" trick, with which PGNs can be efficiently trained in a latent space but deployed in RGB input space for inference.

Figures (8)

• Figure 1: We propose the Prompt Generation Network (PGN), a simple yet effective method that generates prompts conditioned on the input images that benefits the domain adaptation process, while keeping the whole pretrained transformer frozen. (a): the overall learning pipeline including PGN, denoted by $f_{\theta,\mathbf{L}}$: the learned prompt vectors are fed into the pretrained transformer encoder together with image patches for the task or domain of interest. (b): the detailed structure of PGN, a lightweight neural network $g_\theta$ learns probability distributions to select prompt vectors from a Token Library $\mathbf{L}$.
• Figure 2: Prompt Inversion for visual prompting. At inference time, PGN prompts can be mapped to the input space to completely separate the adaptation step from the model's internal computation. In this instance, the PGN outputs 14 prompts which can be visualized as 2 rows containing 7 prompt-patches each.
• Figure 3: Feature similarities. We show the mutual information between the components and the ground-truth labels (details in the supplementary materials). PGN learns very different embeddings to CLIP but when combined achieves strong alignment with the ground-truth labels.
• Figure 4: Parameter efficiency. We vary the number of generated tokens. A Token Library achieves higher performances at significantly less parameters compared to obtaining prompts directly. The library size used here is $8\times$ the number of prompts.
• Figure 5: Automatic domain discovery.$t$-SNE visualisation of PGN outputs. PGN trained on a mixture of datasets automatically allocates the tokens in a manner that recovers the individual training domains.