Learning Causal Domain-Invariant Temporal Dynamics for Few-Shot Action Recognition

TL;DR

CDTD tackles the challenge of few-shot action recognition under domain shift by learning domain-invariant temporal dynamics via causal representation learning in an unsupervised stage, then adapting efficiently to novel data with minimal parameter updates. The method jointly models latent dynamic and domain variables within a VAE-like framework, using a flow-based temporal-transition prior and a domain estimator to capture distributional differences. Phase-wise training—unsupervised discovery of invariant dynamics followed by supervised adaptation—yields state-of-the-art results on multiple benchmarks with high parameter efficiency. The approach is significant for robust action recognition in data-scarce and shifting environments, though theoretical generalization bounds remain an open question for future work.

Abstract

Few-shot action recognition aims at quickly adapting a pre-trained model to the novel data with a distribution shift using only a limited number of samples. Key challenges include how to identify and leverage the transferable knowledge learned by the pre-trained model. We therefore propose CDTD, or Causal Domain-Invariant Temporal Dynamics for knowledge transfer. To identify the temporally invariant and variant representations, we employ the causal representation learning methods for unsupervised pertaining, and then tune the classifier with supervisions in next stage. Specifically, we assume the domain information can be well estimated and the pre-trained image decoder and transition models can be well transferred. During adaptation, we fix the transferable temporal dynamics and update the image encoder and domain estimator. The efficacy of our approach is revealed by the superior accuracy of CDTD over leading alternatives across standard few-shot action recognition datasets.

Figures (8)

• Figure 1: In the few-shot learning setting, what aspects can be effectively transferred from the base data to the novel data? Despite the different temporal dynamics in these two videos, the underlying physical laws are domain-invariant.
• Figure 2: Represent-predict pipeline. (a) We first learn feature representation unsupervisedly for video frames and then supervisedly train the action prediction model. (b) In the "Represent" stage, we design a temporal dynamics model where parameters ($\phi$, $\gamma$) are domain-invariant.
• Figure 3: Our CDTD model for the representation learning stage. (a) During phase 1 on the base data, we train the whole causal temporal dynamics model. (b) During phase 2 on the novel data, we fine-tune both the image encoder and the domain estimator while freezing the image encoder and the temporal dynamicsOur CDTD model for the representation learning stage. (a) During phase 1 on the base data, we train the whole causal temporal dynamics model. (b) During phase 2 on the novel data, we fine-tune both the image encoder and the domain estimator while freezing the image encoder and the temporal dynamics transition module trained during phase 1. transition module trained during phase 1.
• Figure 4: Comparing performance of CDTD$_{NCE}$ (blue) against VL-Prompting (orange) across all action classes on the Sth-Else dataset.
• Figure 5: The comparison of the parameters that need to be updated during adapting on Sth-Else dataset.