TSM: Temporal Shift Module for Efficient Video Understanding

TL;DR

TSM presents a hardware-friendly approach to video understanding by embedding a Temporal Shift Module into 2D CNNs, shifting a small portion of channels across the temporal dimension to fuse information from neighboring frames. By using partial shifts and placing the shift inside residual blocks, it preserves spatial feature learning while enabling powerful temporal modeling, achieving near-3D-CNN performance with 2D-CNN complexity. The method supports offline bi-directional temporal fusion and online uni-directional streaming, delivering strong accuracy, low latency, and edge-deployment viability with substantial throughput gains. Empirically, TSM sets state-of-the-art results on temporally focused datasets (Something-Something variants), shows favorable accuracy-FLOPs trade-offs against ECO and I3D families, and demonstrates practical real-time and edge capabilities, including online video detection improvements. The work provides open-source code and highlights the practical impact of hardware-friendly temporal modeling for scalable video understanding.

Abstract

The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: https://github.com/mit-han-lab/temporal-shift-module.

Figures (8)

• Figure 1: Temporal Shift Module (TSM) performs efficient temporal modeling by moving the feature map along the temporal dimension. It is computationally free on top of a 2D convolution, but achieves strong temporal modeling ability. TSM efficiently supports both offline and online video recognition. Bi-directional TSM mingles both past and future frames with the current frame, which is suitable for high-throughput offline video recognition. Uni-directional TSM mingles only the past frame with the current frame, which is suitable for low-latency online video recognition.
• Figure 2: (a) Latency overhead of TSM due to data movement. (b) Residual TSM achieve better performance than in-place shift. We choose 1/4 proportion residual shift as our default setting. It achieves higher accuracy with a negligible overhead.
• Figure 3: Residual shift is better than in-place shift. In-place shift happens before a convolution layer (or a residual block). Residual shift fuses temporal information inside a residual branch.
• Figure 4: Uni-directional TSM for online video recognition.
• Figure 5: TSM enjoys better accuracy-cost trade-off than I3D family and ECO family on Something-Something-V1 goyal2017something dataset. (GCN includes the cost of ResNet-50 RPN to generate region proposals.)