Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

TL;DR

The paper addresses the challenge of recognizing actions in videos by modeling long-range temporal structure under limited data. It introduces Temporal Segment Networks (TSN), a sparse, video-level framework that aggregates information from short snippets spanning an entire video, enabling end-to-end learning with deep architectures. The study provides a set of practical training recommendations (cross-modality pre-training, partial BN with dropout, data augmentation) and explores multiple input modalities (RGB, RGB difference, optical flow, warped flow). Empirical results on HMDB51 and UCF101 show state-of-the-art performance and qualitative insights from model visualization, demonstrating the effectiveness and efficiency of TSN for video action recognition and its applicability in real-world settings.

Abstract

Deep convolutional networks have achieved great success for visual recognition in still images. However, for action recognition in videos, the advantage over traditional methods is not so evident. This paper aims to discover the principles to design effective ConvNet architectures for action recognition in videos and learn these models given limited training samples. Our first contribution is temporal segment network (TSN), a novel framework for video-based action recognition. which is based on the idea of long-range temporal structure modeling. It combines a sparse temporal sampling strategy and video-level supervision to enable efficient and effective learning using the whole action video. The other contribution is our study on a series of good practices in learning ConvNets on video data with the help of temporal segment network. Our approach obtains the state-the-of-art performance on the datasets of HMDB51 ( $ 69.4\% $) and UCF101 ($ 94.2\% $). We also visualize the learned ConvNet models, which qualitatively demonstrates the effectiveness of temporal segment network and the proposed good practices.

Figures (3)

• Figure 1: Temporal segment network: One input video is divided into $K$ segments and a short snippet is randomly selected from each segment. The class scores of different snippets are fused by an the segmental consensus function to yield segmental consensus, which is a video-level prediction. Predictions from all modalities are then fused to produce the final prediction. ConvNets on all snippets share parameters.
• Figure 2: Examples of four types of input modality: RGB images, RGB difference, optical flow fields (x,y directions), and warped optical flow fields (x,y directions)
• Figure 3: Visualization of ConvNet models for action recognition using DeepDraw DeepDraw. We compare three settings: (1) without pre-train; (2) with pre-train; (3) with temporal segment network . For spatial ConvNets, we plot three generated visualization as color images. For temporal ConvNets, we plot the flow maps of $x$ (left) and $y$ (right) directions in gray-scales. Note all these images are generated from purely random pixels.