Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding

TL;DR

A novel framework, designed to unify and enhance both action detection and anticipation tasks, and which models the mutual influence between intentions and past as well as current information through cross-temporal interactions, thereby refining present and future features and ultimately realizing simultaneous action detection and anticipation.

Abstract

Action understanding, encompassing action detection and anticipation, plays a crucial role in numerous practical applications. However, untrimmed videos are often characterized by substantial redundant information and noise. Moreover, in modeling action understanding, the influence of the agent's intention on the action is often overlooked. Motivated by these issues, we propose a novel framework called the State-Specific Model (SSM), designed to unify and enhance both action detection and anticipation tasks. In the proposed framework, the Critical State-Based Memory Compression module compresses frame sequences into critical states, reducing information redundancy. The Action Pattern Learning module constructs a state-transition graph with multi-dimensional edges to model action dynamics in complex scenarios, on the basis of which potential future cues can be generated to represent intention. Furthermore, our Cross-Temporal Interaction module models the mutual influence between intentions and past as well as current information through cross-temporal interactions, thereby refining present and future features and ultimately realizing simultaneous action detection and anticipation. Extensive experiments on multiple benchmark datasets -- including EPIC-Kitchens-100, THUMOS'14, TVSeries, and the introduced Parkinson's Disease Mouse Behaviour (PDMB) dataset -- demonstrate the superior performance of our proposed framework compared to other state-of-the-art approaches. These results highlight the importance of action dynamics learning and cross-temporal interactions, laying a foundation for future action understanding research.

Figures (9)

• Figure 1: Comparison between Memory-Based and State-Based Methods: (a) Memory-based methods rely on learning effective representations from the entire sequence, which inevitably increases the risk of interference from redundant information and noise. (BG denotes background) (b) Our state-based method constructs ST graph to represent action dynamics. This design encourages the model to focus on the underlying dependencies between actions while suppressing the influence of redundant information and noise.
• Figure 2: Overview of the proposed State-Specific Model. (a) Critical State-Based Memory Compression. Video sequence features are compressed into critical states. (b) Action Pattern Learning. A ST graph is constructed based on critical states to capture action dynamics, and subsequently, a Gated Graph Convolutional Network (Gated GCN) generates potential future cues from the ST graph. (c) Cross-Temporal Interaction. Temporal features interact across different time domains to update current and future features, supporting action detection and anticipation.
• Figure 3: Illustration of State-Transition Graph construction in the APL module.
• Figure 4: CTI simulates the interaction between the intention (potential future cue) and both present and past action information through cross-temporal interaction. This process further refines the present cue and future cue, thereby enhancing support for action detection and anticipation.
• Figure 5: Ablation Experiments. We conduct detailed ablation on (a): Memory Sequence Length, (b): Cluster Number and (c): Shared Classifier, FS and US denote fully shared classifier and unshared classifier, separately.