Efficient Transfer Learning for Video-language Foundation Models

TL;DR

The paper addresses the challenge of efficiently transferring pre-trained video-language foundations to downstream video tasks without forgetting generalizable knowledge. It introduces a parameter-efficient Multi-modal Spatio-Temporal Adapter (MSTA) that adds high-layer adapters for both vision and text, plus a shared projection to align modalities and video-specific up-projections for spatiotemporal adaptation, while freezing the backbone. A spatio-temporal description-guided consistency constraint leverages an LLM to generate descriptors and imposes a cosine-consistency loss, combined with a supervised loss, to maintain generalization with limited data: $\mathcal{L} = \mathcal{L}_{\rm CE} + \alpha \mathcal{L}_{\rm CC}$. Across six datasets and multiple evaluation regimes (zero-shot, few-shot, base-to-novel, and fully supervised), the approach achieves state-of-the-art or competitive results using only 2-7% of trainable parameters, demonstrating strong generalization and practical efficiency for video-language understanding.

Abstract

Pre-trained vision-language models provide a robust foundation for efficient transfer learning across various downstream tasks. In the field of video action recognition, mainstream approaches often introduce additional modules to capture temporal information. Although the additional modules increase the capacity of model, enabling it to better capture video-specific inductive biases, existing methods typically introduce a substantial number of new parameters and are prone to catastrophic forgetting of previously acquired generalizable knowledge. In this paper, we propose a parameter-efficient Multi-modal Spatio-Temporal Adapter (MSTA) to enhance the alignment between textual and visual representations, achieving a balance between generalizable knowledge and task-specific adaptation. Furthermore, to mitigate over-fitting and enhance generalizability, we introduce a spatio-temporal description-guided consistency constraint.This constraint involves providing template inputs (e.g., "a video of \{\textbf{cls}\}") to the trainable language branch and LLM-generated spatio-temporal descriptions to the pre-trained language branch, enforcing output consistency between the branches. This approach reduces overfitting to downstream tasks and enhances the distinguishability of the trainable branch within the spatio-temporal semantic space. We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-supervised learning. Compared to many state-of-the-art methods, our MSTA achieves outstanding performance across all evaluations, while using only 2-7\% of the trainable parameters in the original model.

Figures (3)

• Figure 1: We compared the number of trainable parameters of our method with other state-of-the-art methods, as well as the average Harmonic Mean Performance on the base-to-novel generalization task across four datasets. It can be observed that our method uses fewer parameters while achieving state-of-the-art results.
• Figure 2: The proposed Multi-modal Spatio-Temporal Adapter (MSTA) is designed for transformer-based multi-modal foundation models. It optimizes only the additional adapters, keeping the pretrained CLIP model frozen. To balance discrimination and generalization, we selectively fine-tune a few higher layers ($\geq k$). MSTA employs shared weights for video and text representations, enabling the model to capture shared cues from both modalities.
• Figure 3: Overview of the proposed Multi-modal Spatio-Temporal Adapter (MSTA).