Video Self-Stitching Graph Network for Temporal Action Localization

TL;DR

The paper tackles temporal action localization under large duration variability, with a focus on improving short-action detection. It introduces Video Self-Stitching Graph Network (VSGN), which combines a video self-stitching (VSS) module that creates larger-scale short clips by temporally magnifying brief segments and stitching them with the original clips, with a cross-scale graph pyramid network (xGPN) that propagates information across scales via a pyramid of cross-scale graph networks. Four modules in the scoring/localization stage (M_loc, M_cls, M_adj, M_scr) enable precise localization and classification, trained with a multi-task loss and inference that integrates predictions from both original and magnified clips. Experimental results on THUMOS-14 and ActivityNet-v1.3 demonstrate state-of-the-art performance, with pronounced gains on short actions and improved overall TAL, highlighting the practical potential of cross-scale graph-based temporal models and cross-scale data augmentation for TAL.

Abstract

Temporal action localization (TAL) in videos is a challenging task, especially due to the large variation in action temporal scales. Short actions usually occupy a major proportion in the datasets, but tend to have the lowest performance. In this paper, we confront the challenge of short actions and propose a multi-level cross-scale solution dubbed as video self-stitching graph network (VSGN). We have two key components in VSGN: video self-stitching (VSS) and cross-scale graph pyramid network (xGPN). In VSS, we focus on a short period of a video and magnify it along the temporal dimension to obtain a larger scale. We stitch the original clip and its magnified counterpart in one input sequence to take advantage of the complementary properties of both scales. The xGPN component further exploits the cross-scale correlations by a pyramid of cross-scale graph networks, each containing a hybrid module to aggregate features from across scales as well as within the same scale. Our VSGN not only enhances the feature representations, but also generates more positive anchors for short actions and more short training samples. Experiments demonstrate that VSGN obviously improves the localization performance of short actions as well as achieving the state-of-the-art overall performance on THUMOS-14 and ActivityNet-v1.3.

Figures (7)

• Figure 1: Short actions are the majority in numbers, but have the lowest performance. a) Distribution of action duration in ActivityNet-v1.3 caba2015activitynet. Actions are divided into five duration groups (in seconds): XS (0, 30], S (30, 60], M (60, 120], L (120, 180], and XL (180, inf). b) TAL Performance of different methods on actions of different duration.
• Figure 2: Architecture of the proposed video self-stitching graph network (VSGN). Its takes a video sequence and generates detected actions with start/end time as well as their categories. It has three components: video self-stitching (VSS), cross-scale graph pyramid network (xGPN), and scoring and localization (SoL). VSS (red dashed box, see Fig. \ref{['fig:stitching']} for details) contains four steps to prepare a video sequence as xGPN input. xGPN is composed of multi-level encoder and decoder pyramids. The encoder aggregates features in different levels via a stack of cross-scale graph networks (xGN) (yellow trapezoid area, see Fig. \ref{['fig:feature_aggregation']} for details); the decoder restores the temporal resolution and generates multi-level features for detection. SoL (blue dashed box) contains four modules, the top two predicting action scores and boundaries, the bottom two producing supplementary scores and adjusting boundaries.
• Figure 3: Video self-stitching (VSS). a) Snippet-level features are extracted for the entire video. b) Long video is cut into multiple short clips. c) Each video clip is up-scaled along the temporal dimension. d) Original clip (green dots) and up-scaled clip (orange dots) are stitched into one feature sequence with a gap.
• Figure 4: Cross-scale graph network (xGN). Top: temporal branch; bottom: graph branch. The two branches are fused by addition, followed by an activation function and pooling. Each dot represents a feature, green dots from Clip O and orange dots from Clip U. In the graph branch, the blue arrows represent free edges and the purple arrow represents a cross-scale edge.
• Figure 5: Performance at different temporal scales in terms of accumulated mAP. For ActivityNet, We divide actions into 5 groups based on their duration in seconds: XS (0, 30], S (30, 60], M (60, 120], L (120, 180], and XL (180, inf) . For THUMOS, considering most of its actions fall into the shortest group based on the division above, we further explore the short actions by considering even finer division: XS (0, 3], S (3, 6], M (6, 12], L (12, 18], and XL (18, inf). These curves are obtained by DETAD analysis Alwassel2018DiagnosingEI on the detection results of each method. Our VSGN obviously outperforms the other methods at the shortest duration while maintaining a high rank for longer.