Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling

TL;DR

Attention Prompt Tuning is introduced - a computationally efficient variant of prompt tuning for video-based applications such as action recognition that greatly reduces the number of FLOPs and latency while achieving a significant performance boost over the existing parameter-efficient tuning methods on UCF101, HMDB51, and SSv2 datasets for action recognition.

Abstract

In this paper, we introduce Attention Prompt Tuning (APT) - a computationally efficient variant of prompt tuning for video-based applications such as action recognition. Prompt tuning approaches involve injecting a set of learnable prompts along with data tokens during fine-tuning while keeping the backbone frozen. This approach greatly reduces the number of learnable parameters compared to full tuning. For image-based downstream tasks, normally a couple of learnable prompts achieve results close to those of full tuning. However, videos, which contain more complex spatiotemporal information, require hundreds of tunable prompts to achieve reasonably good results. This reduces the parameter efficiency observed in images and significantly increases latency and the number of floating-point operations (FLOPs) during inference. To tackle these issues, we directly inject the prompts into the keys and values of the non-local attention mechanism within the transformer block. Additionally, we introduce a novel prompt reparameterization technique to make APT more robust against hyperparameter selection. The proposed APT approach greatly reduces the number of FLOPs and latency while achieving a significant performance boost over the existing parameter-efficient tuning methods on UCF101, HMDB51, and SSv2 datasets for action recognition. The code and pre-trained models are available at https://github.com/wgcban/apt

Figures (8)

• Figure 1: Comparison of our Attention Prompt Tuning (APT) for videos action classification with other existing tuning methods: linear probing, adapter tuning chen_adaptformer_2022, visual prompt tuning (VPT) jia_visual_2022, and full fine-tuning.
• Figure 2: The proposed APT architecture. Modules in cyan color are frozen and not updated during fine-tuning. Modules in purple color are learned and updated during fine-tuning. Learnable prompts ($K_p$ and $V_p$) are directly injected into the MHA by concatenating them with the Keys ($K_x$) and Values ($V_x$), while the Queries ($Q_x$) remain unchanged.
• Figure 3: Training loss comparison: with and without prompt reparameterization. The use of scaled reparameterization results in faster convergence and a lower training loss.
• Figure 4: Effect of dropout on attention prompts. Applying 10% dropout to attention prompts results in the best performance on both validation and test sets.
• Figure 5: Effect of random augmentations on training APT. Higher magnitude of augmentations in training results in lower action recognition performance while light / no augmentations results in good action recognition performance.