ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning

TL;DR

The paper tackles the challenge of adapting large image foundation models to video understanding in a parameter-efficient way. It introduces the Spatio-Temporal Adapter (ST-Adapter), a compact bottleneck module that incorporates a depth-wise 3D convolution to enable spatio-temporal reasoning within pretrained image transformers. Across multiple video action recognition benchmarks and pretraining regimes, ST-Adapter matches or exceeds full fine-tuning performance while updating only a small fraction of parameters, demonstrating strong cross-modality transfer with reduced compute. This approach offers a practical path to deploying powerful image models for video tasks with greater efficiency and scalability.

Abstract

Capitalizing on large pre-trained models for various downstream tasks of interest have recently emerged with promising performance. Due to the ever-growing model size, the standard full fine-tuning based task adaptation strategy becomes prohibitively costly in terms of model training and storage. This has led to a new research direction in parameter-efficient transfer learning. However, existing attempts typically focus on downstream tasks from the same modality (e.g., image understanding) of the pre-trained model. This creates a limit because in some specific modalities, (e.g., video understanding) such a strong pre-trained model with sufficient knowledge is less or not available. In this work, we investigate such a novel cross-modality transfer learning setting, namely parameter-efficient image-to-video transfer learning. To solve this problem, we propose a new Spatio-Temporal Adapter (ST-Adapter) for parameter-efficient fine-tuning per video task. With a built-in spatio-temporal reasoning capability in a compact design, ST-Adapter enables a pre-trained image model without temporal knowledge to reason about dynamic video content at a small (~8%) per-task parameter cost, requiring approximately 20 times fewer updated parameters compared to previous work. Extensive experiments on video action recognition tasks show that our ST-Adapter can match or even outperform the strong full fine-tuning strategy and state-of-the-art video models, whilst enjoying the advantage of parameter efficiency. The code and model are available at https://github.com/linziyi96/st-adapter

Figures (3)

• Figure 1: Image-to-video transfer learning strategies. (a) The state-of-the-art methods for adapting a pre-trained image model (e.g., ViT vit in this example) to video tasks (e.g., action recognition) usually adopt the paradigm of first designing a temporal learning module and then fine-tuning the whole network fully vivittimesformerxvit. This is parameter-inefficient since a specific instance of such a large model is resulted for each downstream task. In contrast, (b) we propose to only train a lightweight Spatio-Temporal Adapter with much fewer parameters for each individual downstream task at a significantly smaller computational cost. Surprisingly, our method can match or even surpasses the full fine-tuning based methods (including prior art video models in terms of accuracy), whist enjoying higher parameter efficiency and cheaper training cost.
• Figure 2: Ablation study on efficiency (a) Parameter efficiency: ST-Adapter (with different bottleneck width) is compared with efficient fine-tuning methods in Table \ref{['tab:compare_efficient_transfer']}. (b) Training efficiency: We compare ST-Adapter with Full fine-tuning under different training schedules. Batch size is aligned and their original schedules are shortened proportionally. (c) Data efficiency: Performance comparison on different training data scales. The same ViT-B/16 with CLIP pre-training is used for all experiments.
• Figure 3: Visualization of attention map before and after ST-Adaptation.