TAMT: Temporal-Aware Model Tuning for Cross-Domain Few-Shot Action Recognition

TL;DR

This work tackles cross-domain few-shot action recognition (CDFSAR) by addressing the inefficiency of joint source-target training. It introduces Temporal-Aware Model Tuning (TAMT), a decoupled paradigm that pre-trains on a source domain with self-supervised and supervised objectives and then finely tunes on target domains using a Hierarchical Temporal Tuning Network (HTTN). HTTN combines local Temporal-Aware Adapters (TAA) and a Global Temporal-aware Moment Tuning (GTMT) with Efficient Long-Short Temporal Covariance (ELSTC) to recalibrate features and produce powerful video representations while keeping the backbone frozen. Empirically, TAMT yields 13–31% improvements over state-of-the-art CDFSAR methods and substantially reduces training cost, establishing a strong, efficient baseline for cross-domain few-shot video understanding.

Abstract

Going beyond few-shot action recognition (FSAR), cross-domain FSAR (CDFSAR) has attracted recent research interests by solving the domain gap lying in source-to-target transfer learning. Existing CDFSAR methods mainly focus on joint training of source and target data to mitigate the side effect of domain gap. However, such kind of methods suffer from two limitations: First, pair-wise joint training requires retraining deep models in case of one source data and multiple target ones, which incurs heavy computation cost, especially for large source and small target data. Second, pre-trained models after joint training are adopted to target domain in a straightforward manner, hardly taking full potential of pre-trained models and then limiting recognition performance. To overcome above limitations, this paper proposes a simple yet effective baseline, namely Temporal-Aware Model Tuning (TAMT) for CDFSAR. Specifically, our TAMT involves a decoupled paradigm by performing pre-training on source data and fine-tuning target data, which avoids retraining for multiple target data with single source. To effectively and efficiently explore the potential of pre-trained models in transferring to target domain, our TAMT proposes a Hierarchical Temporal Tuning Network (HTTN), whose core involves local temporal-aware adapters (TAA) and a global temporal-aware moment tuning (GTMT). Particularly, TAA learns few parameters to recalibrate the intermediate features of frozen pre-trained models, enabling efficient adaptation to target domains. Furthermore, GTMT helps to generate powerful video representations, improving match performance on the target domain. Experiments on several widely used video benchmarks show our TAMT outperforms the recently proposed counterparts by 13%$\sim$31%, achieving new state-of-the-art CDFSAR results.

Figures (5)

• Figure 1: (a) Comparison of existing CDFSAR methods in terms of training paradigm under the case of a single source data $\mathcal{S}_{CD}$ and $B$ target data $\{\mathcal{T}_{CD}^{1},\cdots,\mathcal{T}_{CD}^{B}\}$. (b) Comparison (%) of existing CDFSAR methods with K-100 as source data. All results are conducted with $112\times 112$ resolution except methods marked by * ($224\times 224$ resolution)
• Figure 2: (a) Overview of our TAMT paradigm, which pre-trains the models on $\mathcal{S}_{CD}$ and fine-tunes them on $\mathcal{T}_{CD}$. Specifically, for pre-training stage, the model is first optimized with a reconstruction-based SSL solution, while the encoder $\mathcal{E}$ is post-trained with the SL objective. Subsequently, the pre-trained $\mathcal{E}$ is fine-tuned for few-shot adaptation on $\mathcal{T}_{CD}$ by using our HTTN. (b) HTTN for few-shot adaptation, where a metric-based is used for few-shot adaptation. Particularly, our HTTN consists of local Temporal-Aware Adapters (TAA) and Global Temporal-aware Moment Tuning (GTMT).
• Figure 3: Overview of our proposed Hierarchical Temporal Tuning Network (HTTN), where (a) local temporal-aware adapters (TAA) are inserted into the last $L$ transformer blocks to recalibrate the intermediate features of frozen pre-training models in an efficient manner. At the end of HTTN, a Global Temporal-aware Moment Tuning (GTMT) module with efficient long-short temporal covariance (ELSTC) is used to obtain powerful video representations for improving matching performance.
• Figure 4: Convergence curves of FFT and TAMT on SSV2 dataset.
• Figure S5: Feature visualization on setting of K-400 $\rightarrow$ RareAct.