Online Temporal Action Localization with Memory-Augmented Transformer

TL;DR

This work addresses online temporal action localization in streaming videos by introducing MATR, a memory-augmented transformer that leverages a memory queue to incorporate long-term context without strong dependence on fixed input segment length. The method decouples action localization into end and start tasks using two dedicated Transformer decoders and employs distinct class and boundary queries, with memory features guiding start localization. Training uses a holistic, end-to-end objective that combines classification, boundary regression, DIoU-based instance alignment, and a flag-based memory update mechanism, achieving state-of-the-art On-TAL performance on THUMOS14 and MUSES and approaching offline TAL performance. The approach offers practical impact by enabling accurate, online boundary detection with reduced hyperparameter sensitivity, and its memory-based framework suggests robustness for long-term action localization in real-world streaming scenarios.

Abstract

Online temporal action localization (On-TAL) is the task of identifying multiple action instances given a streaming video. Since existing methods take as input only a video segment of fixed size per iteration, they are limited in considering long-term context and require tuning the segment size carefully. To overcome these limitations, we propose memory-augmented transformer (MATR). MATR utilizes the memory queue that selectively preserves the past segment features, allowing to leverage long-term context for inference. We also propose a novel action localization method that observes the current input segment to predict the end time of the ongoing action and accesses the memory queue to estimate the start time of the action. Our method outperformed existing methods on two datasets, THUMOS14 and MUSES, surpassing not only TAL methods in the online setting but also some offline TAL methods.

Figures (10)

• Figure 1: A conceptual illustration of our model. Our model selectively stores information about previously occurred actions in the memory queue from streaming video. When the end of the action is detected in the current input segment, the model retrieves information from the memory queue to locate the start of the action.
• Figure 2: Overall architecture of MATR. MATR consists of four parts: feature extractor, memory-augmented video encoder, instance decoding module, and prediction heads.
• Figure 2: Ablation of the proposed modules on THUMOS14 dataset. The results are reported in mAP measure (%).
• Figure 3: Detailed architecture of the instance decoding module.
• Figure 4: Average mAP (%) versus the segment size in THUMOS14. Red line represents the result of our model and blue line represents the result of OAT-OSN.