Prompt-guided Disentangled Representation for Action Recognition

TL;DR

Prompt-guided Disentangled Representation for Action Recognition (ProDA) introduces a disentangled, action-centric framework for multi-action videos using Spatio-temporal Scene Graphs. It leverages a Dynamic Prompt Module (DPM) and a Video Graph Parsing Neural Network (VGPNN) to split scene graphs into specified-action and remaining-action subgraphs, guided by action prompts derived from multi-hot specifications. A temporal normalization (VGNorm) and an Action Disentanglement Loss (AD Loss) ensure stable, interpretable disentanglement with a reconstruction constraint to preserve essential information. Empirical results on Charades and SportsHHI show state-of-the-art performance and robust action localization cues without explicit localization supervision, highlighting improved modeling of object interactions in complex scenes.

Abstract

Action recognition is a fundamental task in video understanding. Existing methods typically extract unified features to process all actions in one video, which makes it challenging to model the interactions between different objects in multi-action scenarios. To alleviate this issue, we explore disentangling any specified actions from complex scenes as an effective solution. In this paper, we propose Prompt-guided Disentangled Representation for Action Recognition (ProDA), a novel framework that disentangles any specified actions from a multi-action scene. ProDA leverages Spatio-temporal Scene Graphs (SSGs) and introduces Dynamic Prompt Module (DPM) to guide a Graph Parsing Neural Network (GPNN) in generating action-specific representations. Furthermore, we design a video-adapted GPNN that aggregates information using dynamic weights. Experiments in video action recognition demonstrate the effectiveness of our approach when compared with the state-of-the-art methods. Our code can be found in https://github.com/iamsnaping/ProDA.git

Figures (11)

• Figure 1: Comparison between (a) our proposed Prompt-guided Representation Disentanglement method and (b) existing methods using SSG. Existing methods treat action modeling as a feature extraction process, handling all actions in one video. In contrast, our method disentangles the action representation into two parts: one for the specified action and the other for the remaining actions, enabling more focused analysis and better action understanding.
• Figure 2: An overview of our ProDA. We first convert the input video into a sequence of scene graphs, which are then processed by a Video Encoder. Next, we generate two complementary prompts from the Action Set (SAP and UAP) using DPM to guide the VGPNN in disentangling the actions into two complementary SSGs, each containing distinct parts of the action. These disentangled SSGs, along with their fused version, are fed into corresponding readout blocks to obtain the global representations of partial and complete actions, respectively.
• Figure 3: Frame-level weights $s$ predicted by the readout function for different non-distractor-injected SAPs within the same video. The heatmaps (colored bars) visualize the predicted weights, reflecting the relevance of each frame to the target action. The red bars indicate the ground-truth action segments, while the green bars denote our localization results obtained by thresholding $s$ at 0.7.
• Figure 4: t-SNE visualization of disentangled action features across 157 classes on Charades charades. (a) Training set. (b) Test set.
• Figure 5: Our video encoder builds upon Dual-AI han2022dual by introducing additional spatial-spatial and temporal-temporal streams.