Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning

TL;DR

This work tackles the inefficiency of traditional vision-language fine-tuning that relies on extending LM input length with visual prompts. It proposes Memory-Space Visual Prompting (MemVP), which injects visual knowledge directly into the FFN memory of pre-trained language models by augmenting FFN weight matrices with memory-augmented visual entries derived from a learned projector and position embeddings. The approach achieves notable gains in training and inference speed while maintaining or improving VL task performance across BART, T5, and LLaMA, and provides a theoretical complexity advantage by reducing input-length-driven FLOPs. MemVP thus offers a practical, parameter-efficient route to adapt large VL backbones to downstream tasks without requiring VL pre-training, with demonstrated benefits across multiple benchmarks and model scales. Limitations include inference-speed advantages that diminish for long text generation, pointing to avenues for further optimization.

Abstract

Current solutions for efficiently constructing large vision-language (VL) models follow a two-step paradigm: projecting the output of pre-trained vision encoders to the input space of pre-trained language models as visual prompts; and then transferring the models to downstream VL tasks via end-to-end parameter-efficient fine-tuning (PEFT). However, this paradigm still exhibits inefficiency since it significantly increases the input length of the language models. In this paper, in contrast to integrating visual prompts into inputs, we regard visual prompts as additional knowledge that facilitates language models in addressing tasks associated with visual information. Motivated by the finding that Feed-Forward Network (FFN) of language models acts as "key-value memory", we introduce a novel approach termed memory-space visual prompting (MemVP), wherein visual prompts are concatenated with the weights of FFN for visual knowledge injection. Experimental results across various VL tasks and language models reveal that MemVP significantly reduces the training time and inference latency of the finetuned VL models and surpasses the performance of previous PEFT methods. Code: https://github.com/JieShibo/MemVP

Figures (6)

• Figure 1: Illustration of PEFT methods using (a) the conventional input-space visual prompting and (b) our memory-space visual prompting. MemVP outperforms previous paradigms in terms of performance, training speed, and inference speed.
• Figure 2: Training and inference time of LLaMA-7B on a single V100. The training process adopts PEFT in which we only tune LoRA modules. The training batch size and inference batch size are 4 and 16, respectively, to maximize utilization of GPU memory. We also highlight the position when the text token length is 64 w/ and w/o input-space visual prompts. The length of visual prompts is 256 as in LLaVA. We fix the output length to 1.
• Figure 3: Overview of the mainstream paradigms to concatenate vision encoder and language model.(a) Concatenating visual prompts with the text tokens as inputs of the language model is not computation-efficient, e.g., LLaVA, VL-Adapter, VL-PET. (b) Using cross-attention layers to incorporate the visual information from visual tokens is not parameter-efficient, e.g., Flamingo, BLIP. (c) Our MemVP injects visual prompts into the FFN blocks of language models, achieving both parameter and computation efficiency.
• Figure 4: Left: Training time, training memory, and inference time of T5-base on VQAv2. The per-GPU batch sizes for training and inference are 64 and 512, respectively. Measured on V100 GPUs. Right: Average score vs. FLOPs of BART-base on the three datasets. The visual prompts of VL-PET are downsampled to reduce the input length.
• Figure 5: Visual knowledge locating. The similarity values between blod text tokens and keys of visual knowledge are averaged over all layers.