Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognition

TL;DR

This work addresses generalizable video action recognition by reframing visual-language grounding with an Align before Adapt (ALT) paradigm. ALT establishes entity-to-region alignments using a region-aware ViT and an offline text corpus, then feeds the aligned entity text embeddings as queries to a transformer-based video adapter to produce a discriminative video representation, preserving the visual-language alignment from CLIP-style models. The approach combines a Gumbel-Softmax grounded alignment, a ToMe-based region token merging, and a cross-attention video adapter to achieve strong performance across fully supervised, zero-shot, and few-shot regimes with markedly lower compute than many baselines. Empirical results on Kinetics-400/600, HMDB-51, UCF-101, and SS-V2 demonstrate competitive accuracy, superior generalization, and favorable efficiency, highlighting the practical impact of grounding complex activity semantics in reusable entity semantics for robust video understanding.

Abstract

Large-scale visual-language pre-trained models have achieved significant success in various video tasks. However, most existing methods follow an "adapt then align" paradigm, which adapts pre-trained image encoders to model video-level representations and utilizes one-hot or text embedding of the action labels for supervision. This paradigm overlooks the challenge of mapping from static images to complicated activity concepts. In this paper, we propose a novel "Align before Adapt" (ALT) paradigm. Prior to adapting to video representation learning, we exploit the entity-to-region alignments for each frame. The alignments are fulfilled by matching the region-aware image embeddings to an offline-constructed text corpus. With the aligned entities, we feed their text embeddings to a transformer-based video adapter as the queries, which can help extract the semantics of the most important entities from a video to a vector. This paradigm reuses the visual-language alignment of VLP during adaptation and tries to explain an action by the underlying entities. This helps understand actions by bridging the gap with complex activity semantics, particularly when facing unfamiliar or unseen categories. ALT demonstrates competitive performance while maintaining remarkably low computational costs. In fully supervised experiments, it achieves 88.1% top-1 accuracy on Kinetics-400 with only 4947 GFLOPs. Moreover, ALT outperforms the previous state-of-the-art methods in both zero-shot and few-shot experiments, emphasizing its superior generalizability across various learning scenarios.

Figures (6)

• Figure 1: Left: Paradigm comparison between traditional adaptation approaches and our "Align before Adapt" method. Right: Zero-shot and few-shot performance comparison on HMDB-51 dataset. Pretrained on Kinetics-400, our method surpasses the previous state of the arts.
• Figure 2: An overview of our framework: we utilize a video clip and an offline text corpus as inputs to learn a video representation, which is supervised with the objective of maximizing the similarity score with the text representation of the corresponding action label.
• Figure 3: Detailed network components: (a) The region-aware image encoder includes a ViT with plug-in token merging modules. MSA, MLP, and LN indicate multi-head self-attention, multilayer perception, and layernorm, respectively. (b) The entity-to-region alignment module obtains the aligned query in a softmax-weight-sum manner. and (c) shows the multi-modal video adapter, with each block containing a stack of hybrid modules composed of attention layers and 1D temporal convolution.
• Figure 4: Left: Visualization of visual-semantic correspondences with the tool Chen_gscoreCAM22. For each row: Column (2) visualizes the visual correspondence to text entities generated by the action label; Column (3) visualizes region-aware embeddings under ToMe; Column (4) and (5) show the two of the fine-grained corresponding visual patterns to specific text entities, which are geometrically consistent with Column (3). Right: Visualization of Accuracy v.s. FLOPs performance.
• Figure 5: Few-shot comparison: we compare ALTs with previous SOTAs on HMDB-51 and UCF-101. All the models are trained on Kinetics-400, with top-1 accuracies(%) reported under a single-view inference.