Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification

TL;DR

The paper tackles the speed-accuracy tradeoff in video classification by systematically dissecting where 3D convolutions add value. Through network-surgery experiments, it shows that replacing many 3D convs with 2D ones—especially at lower layers—can hurt temporal modeling, while top-heavy designs that keep 3D processing at higher, semantically rich levels often yield faster and more accurate results. It further introduces separable spatio-temporal convolutions (S3D) and a spatio-temporal gating mechanism (S3D-G), which together significantly improve efficiency and accuracy across multiple datasets and tasks, including action detection. The findings offer practical design guidelines for building scalable video understanding systems applicable to diverse datasets and real-time settings.

Abstract

Despite the steady progress in video analysis led by the adoption of convolutional neural networks (CNNs), the relative improvement has been less drastic as that in 2D static image classification. Three main challenges exist including spatial (image) feature representation, temporal information representation, and model/computation complexity. It was recently shown by Carreira and Zisserman that 3D CNNs, inflated from 2D networks and pretrained on ImageNet, could be a promising way for spatial and temporal representation learning. However, as for model/computation complexity, 3D CNNs are much more expensive than 2D CNNs and prone to overfit. We seek a balance between speed and accuracy by building an effective and efficient video classification system through systematic exploration of critical network design choices. In particular, we show that it is possible to replace many of the 3D convolutions by low-cost 2D convolutions. Rather surprisingly, best result (in both speed and accuracy) is achieved when replacing the 3D convolutions at the bottom of the network, suggesting that temporal representation learning on high-level semantic features is more useful. Our conclusion generalizes to datasets with very different properties. When combined with several other cost-effective designs including separable spatial/temporal convolution and feature gating, our system results in an effective video classification system that that produces very competitive results on several action classification benchmarks (Kinetics, Something-something, UCF101 and HMDB), as well as two action detection (localization) benchmarks (JHMDB and UCF101-24).

Figures (7)

• Figure 1: Our goal is to classify videos into different categories, as shown in the top row. We focus on two qualitatively different kinds of datasets: Something-something which requires recognizing low-level physical interactions, and Kinetics, which requires recognizing high-level activities. The main question we seek to answer is what kind of network architecture to use. We consider 4 main variants: I2D, which is a 2D CNN, operating on multiple frames; I3D, which is a 3D CNN, convolving over space and time; Bottom-Heavy I3D, which uses 3D in the lower layers, and 2D in the higher layers; and Top-Heavy I3D, which uses 2D in the lower (larger) layers, and 3D in the upper layers.
• Figure 2: Network architecture details for (a) I3D, (b) I2D, (c) Bottom-Heavy and (d) Top-Heavy variants. $K$ indexes the spatio-temporal convolutional layers. The "2D Inc." and "3D Inc." blocks refer to 2D and 3D inception blocks, defined in Figure \ref{['fig:inc_blocks']}.
• Figure 3: \ref{['fig:inc_block_2d']} 2D Inception block; \ref{['fig:inc_block_3d']} 3D Inception block; \ref{['fig:sep_inc_block']} 3D temporal separable Inception block used in S3D networks.
• Figure 4: Accuracy vs number of FLOPS needed to perform inference on 64 RGB frames. Left: Mini-Kinetics-200 dataset. Right: Something-something dataset. Solid lines denote top-heavy models, dotted lines denote bottom-heavy models. Orange denotes spatial and temporal separable 3D convolutions, blue denotes full 3D convolutions.
• Figure 5: Statistics of convolutional filter weights of an I3D model trained on Kinetics-Full. Each boxplot shows the distribution of $W_l(t,:,:,:)$ for temporal offset $t$, with $t=0$ being in the middle. Results for different layers $l$ are shown in different panels, with lowest layers on the left. All filters with different temporal offset are initialized with the same set of weights. Low-level filters essentially ignore the temporal dimension, unlike higher level filters, where the weights distributed nicely across different temporal offsets.