Kronecker Mask and Interpretive Prompts are Language-Action Video Learners
Jingyi Yang, Zitong Yu, Xiuming Ni, Jia He, Hui Li
TL;DR
The paper tackles the challenge of transferring CLIP-like vision-language pretraining to video by jointly addressing temporal modeling and verb-centric language. It introduces CLAVER, which combines Kronecker Mask Attention (KMTA/KMCTA) for expansive, structured spatiotemporal modeling with interpretive prompts generated by LLMs to shift text guidance from nouns to actions. The approach yields substantial improvements on Kinetics-400/600, HMDB-51, and UCF-101 in fully supervised, zero-shot, and few-shot settings, and is shown to yield more interpretable attention patterns that focus on action-relevant regions and verbs. This work demonstrates that aligning dynamic action semantics in both visual and textual branches can significantly enhance video understanding and generalization in practical settings.
Abstract
Contrastive language-image pretraining (CLIP) has significantly advanced image-based vision learning. A pressing topic subsequently arises: how can we effectively adapt CLIP to the video domain? Recent studies have focused on adjusting either the textual or visual branch of CLIP for action recognition. However, we argue that adaptations of both branches are crucial. In this paper, we propose \textbf{CLAVER}: a \textbf{C}ontrastive \textbf{L}anguage-\textbf{A}ction \textbf{V}ideo Learn\textbf{er}, designed to shift CLIP's focus from the alignment of static visual objects and concrete nouns to the alignment of dynamic action behaviors and abstract verbs. Specifically, we introduce a novel Kronecker mask attention for temporal modeling. Our tailored Kronecker mask offers three benefits 1) it expands the temporal receptive field for each token, 2) it serves as an effective spatiotemporal heterogeneity inductive bias, mitigating the issue of spatiotemporal homogenization, and 3) it can be seamlessly plugged into transformer-based models. Regarding the textual branch, we leverage large language models to generate diverse, sentence-level and semantically rich interpretive prompts of actions, which shift the model's focus towards the verb comprehension. Extensive experiments on various benchmarks and learning scenarios demonstrate the superiority and generality of our approach.