End-to-End Streaming Video Temporal Action Segmentation with Reinforce Learning

TL;DR

This work addresses streaming temporal action segmentation (STAS), a online variant of TAS that processes untrimmed videos clip-by-clip. It introduces SVTAS-RL, an end-to-end model that uses a clustering-based observation pipeline and a Hierarchical Block Recurrent Transformer (HBRT) memory to align with the STAS data manifold, coupled with reinforcement learning to mitigate optimization dilemmas. The authors propose two RL strategies, Monte Carlo REINFORCE and Temporal Difference Actor-Critic, to estimate gradients aligned with the full-sequence action integrity objective. Experiments across Breakfast, 50Salads, GTEA, and EGTEA show SVTAS-RL achieving competitive TAS state-of-the-art results and substantial improvements over existing STAS methods, particularly on ultra-long videos like EGTEA. The work demonstrates the effectiveness of clustering-based representations and RL-guided end-to-end optimization for online action segmentation, with practical implications for real-time video understanding.

Abstract

The streaming temporal action segmentation (STAS) task, a supplementary task of temporal action segmentation (TAS), has not received adequate attention in the field of video understanding. Existing TAS methods are constrained to offline scenarios due to their heavy reliance on multimodal features and complete contextual information. The STAS task requires the model to classify each frame of the entire untrimmed video sequence clip by clip in time, thereby extending the applicability of TAS methods to online scenarios. However, directly applying existing TAS methods to SATS tasks results in significantly poor segmentation outcomes. In this paper, we thoroughly analyze the fundamental differences between STAS tasks and TAS tasks, attributing the severe performance degradation when transferring models to model bias and optimization dilemmas. We introduce an end-to-end streaming video temporal action segmentation model with reinforcement learning (SVTAS-RL). The end-to-end modeling method mitigates the modeling bias introduced by the change in task nature and enhances the feasibility of online solutions. Reinforcement learning is utilized to alleviate the optimization dilemma. Through extensive experiments, the SVTAS-RL model significantly outperforms existing STAS models and achieves competitive performance to the state-of-the-art TAS model on multiple datasets under the same evaluation criteria, demonstrating notable advantages on the ultra-long video dataset EGTEA. Code is available at https://github.com/Thinksky5124/SVTAS.

Figures (8)

• Figure 1: The phenomenon of modeling bias in TAS models migrating to STAS task. All visualization manifolds of the data are obtained through t-SNE. In the image, the horizontal axis represents the length of the video clip after cropping, with the complete video on the far right and progressively shorter lengths towards the left.The first row represents the manifolds of the RGB modality of video images, the second row depicts the manifolds of the TAS model sequential features, and the third row illustrates the manifolds of the clustering model features. As the duration of segmented video clips decreases, which means from the TAS task to the STAS task, we can observe: (a) the data manifold of the original video gradually transitions from a distorted line to a clustering swiss roll; (b) The shorter the segmented video clip, the less applicable the sequential model becomes, but clustering model performs the opposite.
• Figure 2: Overview of SVTAS-RL model. To train the SVTAS-RL model, we first sample a video clip $v_j$, which is then parsed by the observation model $\mathbb{H}$ to yield the current state $s_j$. Subsequently, the agent $\mathbb{A}$ makes a decision $a_j$ based on the current state $s_j$, and the decision is evaluated by $Q$ to obtain a reward $r_j$.
• Figure 3: Overview of Hierarchical Block Recurrent Transformer (HBRT).
• Figure 4: The importance of global contextual information. Class Activation Map (CAM) for red frame in full sequence TAS and STAS. Streaming CAM is mostly black because the frames are not available at inference time.
• Figure 5: Comparison of feature manifold. All features are visualized by t-SNE. Obviously, TAS is a sequence-to-sequence transformation paradigm and STAS is a clustering paradigm.