Enhancing Video Transformers for Action Understanding with VLM-aided Training

TL;DR

The Four-tiered Prompts (FTP) framework is proposed that takes advantage of the complementary strengths of ViTs and VLMs but improves the visual encodings to be more comprehensive and general by aligning them with VLM outputs.

Abstract

Owing to their ability to extract relevant spatio-temporal video embeddings, Vision Transformers (ViTs) are currently the best performing models in video action understanding. However, their generalization over domains or datasets is somewhat limited. In contrast, Visual Language Models (VLMs) have demonstrated exceptional generalization performance, but are currently unable to process videos. Consequently, they cannot extract spatio-temporal patterns that are crucial for action understanding. In this paper, we propose the Four-tiered Prompts (FTP) framework that takes advantage of the complementary strengths of ViTs and VLMs. We retain ViTs' strong spatio-temporal representation ability but improve the visual encodings to be more comprehensive and general by aligning them with VLM outputs. The FTP framework adds four feature processors that focus on specific aspects of human action in videos: action category, action components, action description, and context information. The VLMs are only employed during training, and inference incurs a minimal computation cost. Our approach consistently yields state-of-the-art performance. For instance, we achieve remarkable top-1 accuracy of 93.8% on Kinetics-400 and 83.4% on Something-Something V2, surpassing VideoMAEv2 by 2.8% and 2.6%, respectively.

Figures (10)

• Figure 1: Conceptual idea. Different domains potentially emphasize other aspects of an action's performance, here reflected in the different label sets used in benchmark datasets. VLMs can provide relevant details of video content that might be insufficiently covered by a pretrained visual encoder. In the FTP framework, we employ the textual descriptions from four prompts to align a ViT's visual embeddings during training. This way, we generate richer, comprehensive video representations. During inference, we don't require the VLM and benefit from the more general feature representations to improve action understanding across domains.
• Figure 2: Architecture of the FTP framework. (a) Four feature processors attend to different aspects of the video contents. Their outputs are integrated and subsequently classified. (b) Architecture of a feature processor, with spatial and temporal pooling and projection. The outputs are concatenated to produce the output $v_i$.
• Figure 3: Integration process for feature processor and visual encoder outputs. Each feature processor output $v_i$ is replicated, projected, and finally element-wise summed with visual encoder output $m$.
• Figure 4: Training stage 1. Concatenated keyframe images are processed by a VLM using four prompts. The outputs pass through a text encoder to yield text embeddings. The feature processors are trained with contrastive loss to project the visual embeddings onto the text embeddings. VLM, text encoder, and visual encoder are frozen.
• Figure 5: Example classifications for two different combinations of prompts, with softmax outputs. Top row is misclassified, bottom row is correctly classified.