APLA: A Simple Adaptation Method for Vision Transformers

TL;DR

APLA presents a simple yet effective method for adapting Vision Transformers by tuning only a randomly selected subset of columns in the projection layer immediately after the self-attention stage, plus the final classifier. This no-parameter-addition approach achieves state-of-the-art results across 46 datasets and multiple tasks while reducing training memory and time, with no inference cost. The key insight is that the post-attention projection layer $W_O$ is the most impactful component for adaptation, and random low-rank updates to its columns suffice to reweight features across attention heads. Across diverse foundation models and pretraining schemes, APLA consistently outperforms 17 competitive methods, offering a practical and scalable solution for efficient ViT adaptation.

Abstract

Existing adaptation techniques typically require architectural modifications or added parameters, leading to high computational costs and complexity. We introduce Attention Projection Layer Adaptation (APLA), a simple approach to adapt vision transformers (ViTs) without altering the architecture or adding parameters. Through a systematic analysis, we find that the layer immediately after the attention mechanism is crucial for adaptation. By updating only this projection layer, or even just a random subset of this layer's weights, APLA achieves state-of-the-art performance while reducing GPU memory usage by up to 52.63% and training time by up to 43.0%, with no extra cost at inference. Across 46 datasets covering a variety of tasks including scene classification, medical imaging, satellite imaging, and fine-grained classification, APLA consistently outperforms 17 other leading adaptation methods, including full fine-tuning, on classification, segmentation, and detection tasks. The code is available at https://github.com/MoeinSorkhei/APLA.

Figures (9)

• Figure 1: APLA achieves state-of-the-art for ViT adaption. It yields better performance for a given GPU memory budget during training compared to full fine-tuning and leading adaptation methods. Similar savings are observed at inference (see Appendix \ref{['apx:more_exp_results']}).
• Figure 2: Investigating adaptation performance of individual ViT components. We evaluate the adaptation effectiveness of each ViT component in isolation across various downstream tasks, reporting the average performance. Results show that the attention output projection layer ($W_O$), located immediately after the attention mechanism, is the most effective for adaptation. See Section \ref{['sec:choosing']} and Table \ref{['tab:which-component-to-tune']} for detailed results.
• Figure 3: Computational cost. We report the memory footprint of various adaptation methods during training with a batch size of 64. Results for parameter count, memory, and throughput during training and inference are provided in Appendix \ref{['apx:more_exp_results_computes']}.
• Figure 4: Performance vs. compute cost. We compare each method's performance against GPU memory (left) and latency (right) during training across different model capacities.
• Figure 5: Computational costs. We report memory footprint and latency of various adaptation methods during training (top) and inference (middle) for ViT-B with a batch size of 64. Additionally, we provide the number of tunable parameter count for each method (bottom), averaged across all the datasets.