Fine-grained Knowledge Graph-driven Video-Language Learning for Action Recognition

TL;DR

This work tackles coarse-grained video action recognition by injecting a fine-grained action-parsing knowledge graph into a CLIP-based video-language framework. KG-CLIP constructs a multi-modal knowledge graph over videos, actions, and body movements, and uses a Triplet Encoder with relation-specific subspaces plus a deviation compensation mechanism to align cross-modal representations. The model jointly optimizes multi-modal contrastive learning and triplet-based graph learning, with a final inference that fuses both signals: $\mathbf{S}^{output}=\tfrac{1}{2}(\mathbf{S}^{mm}+\mathbf{S}^{tri})$. On Kinetics-TPS, KG-CLIP achieves state-of-the-art accuracy and demonstrates strong data efficiency, particularly with few frames or limited training data, highlighting the practical impact of structured knowledge for video understanding.

Abstract

Recent work has explored video action recognition as a video-text matching problem and several effective methods have been proposed based on large-scale pre-trained vision-language models. However, these approaches primarily operate at a coarse-grained level without the detailed and semantic understanding of action concepts by exploiting fine-grained semantic connections between actions and body movements. To address this gap, we propose a contrastive video-language learning framework guided by a knowledge graph, termed KG-CLIP, which incorporates structured information into the CLIP model in the video domain. Specifically, we construct a multi-modal knowledge graph composed of multi-grained concepts by parsing actions based on compositional learning. By implementing a triplet encoder and deviation compensation to adaptively optimize the margin in the entity distance function, our model aims to improve alignment of entities in the knowledge graph to better suit complex relationship learning. This allows for enhanced video action recognition capabilities by accommodating nuanced associations between graph components. We comprehensively evaluate KG-CLIP on Kinetics-TPS, a large-scale action parsing dataset, demonstrating its effectiveness compared to competitive baselines. Especially, our method excels at action recognition with few sample frames or limited training data, which exhibits excellent data utilization and learning capabilities.

Figures (4)

• Figure 1: An overview of KG-CLIP. (A) Construction of a structured multi-modal knowledge graph via systematic action parsing to encapsulate relationships between video, text and contextual elements. (B) Encoders based on the CLIP model learn embeddings for visual and textual knowledge. The visual encoder leverages both spatial and temporal components to obtain comprehensive video representations. (C) A triplet learning module enables entity alignment by projecting head and tail entities into a relation-specific subspace, and reconciling modality gaps in the multi-modal embedding space to better facilitate action recognition.
• Figure 2: An illustration of modality gap in video-language contrastive learning.
• Figure 3: Comparison of KG-CLIP with baselines supported by body movement prompts on four datasets with varying numbers of video frames and backbones.
• Figure 4: Top-1 accuracy (%) on four datasets with different number of video frames and multi-modal learning settings ($\lambda$) using ViT-B/32 as backbone.