Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AIM: Adapting Image Models for Efficient Video Action Recognition

Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, Mu Li

TL;DR

This work tackles the high computational cost of fully finetuning video transformers by freezing a pre-trained image ViT and introducing lightweight adapters to enable spatial, temporal, and joint spatiotemporal adaptation. The AIM framework demonstrates that high-quality video action recognition can be achieved with far fewer tunable parameters than conventional video models, while remaining compatible with various image pre-trained backbones and even benefiting from CLIP pretraining. Through extensive ablations and cross-dataset comparisons (K400, K700, SSv2, Diving-48), AIM shows competitive or superior performance with tangible gains in data efficiency and training/or inference cost. This approach offers a practical path to leverage powerful image foundation models for video tasks with reduced compute and memory requirements.

Abstract

Recent vision transformer based video models mostly follow the ``image pre-training then finetuning" paradigm and have achieved great success on multiple video benchmarks. However, full finetuning such a video model could be computationally expensive and unnecessary, given the pre-trained image transformer models have demonstrated exceptional transferability. In this work, we propose a novel method to Adapt pre-trained Image Models (AIM) for efficient video understanding. By freezing the pre-trained image model and adding a few lightweight Adapters, we introduce spatial adaptation, temporal adaptation and joint adaptation to gradually equip an image model with spatiotemporal reasoning capability. We show that our proposed AIM can achieve competitive or even better performance than prior arts with substantially fewer tunable parameters on four video action recognition benchmarks. Thanks to its simplicity, our method is also generally applicable to different image pre-trained models, which has the potential to leverage more powerful image foundation models in the future. The project webpage is \url{https://adapt-image-models.github.io/}.

AIM: Adapting Image Models for Efficient Video Action Recognition

TL;DR

This work tackles the high computational cost of fully finetuning video transformers by freezing a pre-trained image ViT and introducing lightweight adapters to enable spatial, temporal, and joint spatiotemporal adaptation. The AIM framework demonstrates that high-quality video action recognition can be achieved with far fewer tunable parameters than conventional video models, while remaining compatible with various image pre-trained backbones and even benefiting from CLIP pretraining. Through extensive ablations and cross-dataset comparisons (K400, K700, SSv2, Diving-48), AIM shows competitive or superior performance with tangible gains in data efficiency and training/or inference cost. This approach offers a practical path to leverage powerful image foundation models for video tasks with reduced compute and memory requirements.

Abstract

Recent vision transformer based video models mostly follow the ``image pre-training then finetuning" paradigm and have achieved great success on multiple video benchmarks. However, full finetuning such a video model could be computationally expensive and unnecessary, given the pre-trained image transformer models have demonstrated exceptional transferability. In this work, we propose a novel method to Adapt pre-trained Image Models (AIM) for efficient video understanding. By freezing the pre-trained image model and adding a few lightweight Adapters, we introduce spatial adaptation, temporal adaptation and joint adaptation to gradually equip an image model with spatiotemporal reasoning capability. We show that our proposed AIM can achieve competitive or even better performance than prior arts with substantially fewer tunable parameters on four video action recognition benchmarks. Thanks to its simplicity, our method is also generally applicable to different image pre-trained models, which has the potential to leverage more powerful image foundation models in the future. The project webpage is \url{https://adapt-image-models.github.io/}.
Paper Structure (24 sections, 5 equations, 9 figures, 6 tables, 1 algorithm)

This paper contains 24 sections, 5 equations, 9 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminary
  5. Spatial Adaptation
  6. Temporal Adaptation
  7. Joint Adaptation
  8. Experiments
  9. Effectiveness of Components
  10. Comparisons to the State of the art
  11. Results on Kinetics-400 and Kinetics-700
  12. Results on Something-Something-v2
  13. Results on Diving-48
  14. Discussion
  15. Conclusion
  16. ...and 9 more sections

Figures (9)

  • Figure 1: Left: Pipeline comparison between traditional full finetuning and our efficient finetuning. Right: Performance comparison on K400 dataset kay2017kinetics. Bubble size indicates GFLOPS at inference time. Our proposed AIM achieves the highest accuracy while enjoying significantly less number of tunable parameters and GFLOPS.
  • Figure 2: We show how we adapt a standard ViT block (b) for video action recognition, by gradually adding spatial adaptation (c), temporal adaptation (d) and joint adaptation (e). Note that S-MSA and T-MSA share weights but are applied to different input dimensions. During training, only newly added Adapters are updated while all the other layers are frozen.
  • Figure 3: Comparisons on Kinetics-700.
  • Figure 4: Comparisons on Diving-48.
  • Figure 5: Data efficiency comparison. AIM outperforms full finetuned TimeSformer under all scenarios, especially in low data regime.
  • ...and 4 more figures