Probabilistic Temporal Masked Attention for Cross-view Online Action Detection

TL;DR

The paper tackles cross-view online action detection by introducing Probabilistic Temporal Masked Attention (PTMA), a two-branch model that combines a variational autoencoder-based probabilistic encoder with a GRU-TMA classifier. The probabilistic branch learns latent representations $z$ to capture view-invariant information, while the classification branch uses temporal masked attention to refine frame-level predictions in an autoregressive fashion. PTMA jointly optimizes a reconstruction loss and a KL-divergence term alongside standard cross-entropy loss, enabling robust cross-view generalization, including to unseen viewpoints. Across cs, cv, and csv evaluation protocols on DAHLIA, IKEA ASM, and Breakfast, PTMA achieves state-of-the-art results, highlighting its effectiveness in mitigating view-specific biases and its practical potential for cross-view OAD tasks.

Abstract

As a critical task in video sequence classification within computer vision, Online Action Detection (OAD) has garnered significant attention. The sensitivity of mainstream OAD models to varying video viewpoints often hampers their generalization when confronted with unseen sources. To address this limitation, we propose a novel Probabilistic Temporal Masked Attention (PTMA) model, which leverages probabilistic modeling to derive latent compressed representations of video frames in a cross-view setting. The PTMA model incorporates a GRU-based temporal masked attention (TMA) cell, which leverages these representations to effectively query the input video sequence, thereby enhancing information interaction and facilitating autoregressive frame-level video analysis. Additionally, multi-view information can be integrated into the probabilistic modeling to facilitate the extraction of view-invariant features. Experiments conducted under three evaluation protocols: cross-subject (cs), cross-view (cv), and cross-subject-view (csv) show that PTMA achieves state-of-the-art performance on the DAHLIA, IKEA ASM, and Breakfast datasets.

Figures (6)

• Figure 1: Illustration of OAD under cross-view settings. The OAD model is trained on $v_1$ while tested on unseen $v_2$ and $v_3$. The training and testing need not contain identical footage.
• Figure 2: Illustration of the proposed PTMA model. (a) PTMA's overview. The upper classification branch includes a GRU-TMA cell for preliminary encoding input features $\mathbf{X}$. The lower probabilistic branch acts as a probabilistic model, mapping $\mathbf{X}$ to latent representations $\mathbf{z}$, which is used to interact with the GRU-TMA cell for temporal masked attention. (b) The expanded view of the GRU-TMA cell. The preliminary encoding $\mathbf{H}$ and the attention output $\mathbf{A}$ are element-wise added to produce the refined temporal encoding $\tilde{\mathbf{H}}$ for classification. (c) Different training modes for PTMA. Various modules of PTMA are activated or deactivated and the base model is a fine-tuned GRU without TMA module.
• Figure 3: Comparison of latent space for probabilistic modeling using single-view or multi-view sources for data reconstruction. The area $S$ is an ideal latent space where the model can learn view-invariant features.
• Figure 4: Illustration of the Temporal Masked Attention. (a) Computation of TMA. The input $f_q(z)$ functions as query ($q$), and the raw temporal encoding $\mathbf{H}$ is key ($k$) and value ($v$). (b) Generation of the temporal mask. The red square denotes the temporal window for OAD, spanning a length of T. The purple area symbolizes an inaccessible future, and the green area indicates a distant long history. During inference, where the temporal window extends over the full video sequence, the inclusion of distant history could hamper the current OAD task. To counteract this, information exchange with latent view-level representation is confined to the red zone, aligning with training constraints. The model is guided to prioritize short-term historical information by masking both future and distant history frames, thereby enhancing task relevance and model focus.
• Figure 5: Illustration of three evaluation mechanisms. $v_i$ and $v_j$ are different viewpoints.