When Spatial meets Temporal in Action Recognition

TL;DR

The Temporal Integration and Motion Enhancement (TIME) layer is introduced, a novel preprocessing technique designed to incorporate temporal information that enhances recognition accuracy and offers valuable insights for video processing tasks.

Abstract

Video action recognition has made significant strides, but challenges remain in effectively using both spatial and temporal information. While existing methods often focus on either spatial features (e.g., object appearance) or temporal dynamics (e.g., motion), they rarely address the need for a comprehensive integration of both. Capturing the rich temporal evolution of video frames, while preserving their spatial details, is crucial for improving accuracy. In this paper, we introduce the Temporal Integration and Motion Enhancement (TIME) layer, a novel preprocessing technique designed to incorporate temporal information. The TIME layer generates new video frames by rearranging the original sequence, preserving temporal order while embedding $N^2$ temporally evolving frames into a single spatial grid of size $N \times N$. This transformation creates new frames that balance both spatial and temporal information, making them compatible with existing video models. When $N=1$, the layer captures rich spatial details, similar to existing methods. As $N$ increases ($N\geq2$), temporal information becomes more prominent, while the spatial information decreases to ensure compatibility with model inputs. We demonstrate the effectiveness of the TIME layer by integrating it into popular action recognition models, such as ResNet-50, Vision Transformer, and Video Masked Autoencoders, for both RGB and depth video data. Our experiments show that the TIME layer enhances recognition accuracy, offering valuable insights for video processing tasks.

Figures (37)

• Figure 1: Variants of the TIME layer. The spatial block arrangement captures long-term motion spatially by incorporating broad visual changes across frames, while it emphasizes short-term motion within each frame's temporal sequence (tube masking, as shown in (a), temporally obscures short-term motion details for VideoMAE). In contrast, the temporal block arrangement records short-term motion frame by frame, yet captures long-term motion temporally across the sequence (tube masking here obscures long-term motion). Each approach provides a unique perspective on balancing spatial and temporal information through the spatial-temporal balance parameter, $N$, in video data.
• Figure 2: Comparison of spatial and temporal block arrangements across RGB and depth modalities using ResNet-50 (pretrained on ImageNet-1K), ViT (pretrained on ImageNet-1K), and VideoMAE (trained from scratch) on three datasets.
• Figure 3: Impact of the spatial-temporal balance parameter $N$ on VideoMAE. For the RGB modality, smaller values of $N$ generally yield better performance, as RGB content tends to be more visually complex. In contrast, the 3D Action Pairs dataset, containing simpler actions, achieves optimal results with a larger $N$ compared to more challenging datasets like HMDB51 and UCF101.
• Figure 4: Impact of the TIME layer on VideoMAE mask ratios. We analyze RGB and depth videos across various datasets, including small-scale and challenging ones like HMDB51. Results show that the TIME layer generally favors a lower mask ratio for optimal performance on RGB videos.
• Figure 5: Per-layer weight similarity comparison between VideoMAE encoders with and without the TIME layer. The models are first trained from scratch, followed by fine-tuning of the encoders. The horizontal axis shows cosine similarity scores (ranging from 0 to 1), where higher values indicate greater similarity between the weights of the models. The vertical axis lists the individual layers within the VideoMAE encoder.