Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Context-Aware Network Based on Multi-scale Spatio-temporal Attention for Action Recognition in Videos

Xiaoyang Li, Wenzhu Yang, Kanglin Wang, Tiebiao Wang, Qingsong Fei

TL;DR

CAN addresses the challenge of multi-scale spatio-temporal cues in video action recognition by introducing two complementary modules: MTCM for multi-scale temporal modeling and GSCM for multi-scale spatial cueing. The architecture enables adaptive fusion of temporal and spatial information through sequential integration (GSCM then MTCM) and four-path grouping with residual connections, yielding robust representations. Empirical results on five benchmarks show competitive or superior accuracy with efficient runtime, and ablations confirm the necessity of multi-scale design and careful integration. The work demonstrates that explicit, multi-scale context integration can significantly improve action recognition without resorting to heavy transformer architectures, making CAN practical for real-world deployment.

Abstract

Action recognition is a critical task in video understanding, requiring the comprehensive capture of spatio-temporal cues across various scales. However, existing methods often overlook the multi-granularity nature of actions. To address this limitation, we introduce the Context-Aware Network (CAN). CAN consists of two core modules: the Multi-scale Temporal Cue Module (MTCM) and the Group Spatial Cue Module (GSCM). MTCM effectively extracts temporal cues at multiple scales, capturing both fast-changing motion details and overall action flow. GSCM, on the other hand, extracts spatial cues at different scales by grouping feature maps and applying specialized extraction methods to each group. Experiments conducted on five benchmark datasets (Something-Something V1 and V2, Diving48, Kinetics-400, and UCF101) demonstrate the effectiveness of CAN. Our approach achieves competitive performance, outperforming most mainstream methods, with accuracies of 50.4% on Something-Something V1, 63.9% on Something-Something V2, 88.4% on Diving48, 74.9% on Kinetics-400, and 86.9% on UCF101. These results highlight the importance of capturing multi-scale spatio-temporal cues for robust action recognition.

Context-Aware Network Based on Multi-scale Spatio-temporal Attention for Action Recognition in Videos

TL;DR

CAN addresses the challenge of multi-scale spatio-temporal cues in video action recognition by introducing two complementary modules: MTCM for multi-scale temporal modeling and GSCM for multi-scale spatial cueing. The architecture enables adaptive fusion of temporal and spatial information through sequential integration (GSCM then MTCM) and four-path grouping with residual connections, yielding robust representations. Empirical results on five benchmarks show competitive or superior accuracy with efficient runtime, and ablations confirm the necessity of multi-scale design and careful integration. The work demonstrates that explicit, multi-scale context integration can significantly improve action recognition without resorting to heavy transformer architectures, making CAN practical for real-world deployment.

Abstract

Action recognition is a critical task in video understanding, requiring the comprehensive capture of spatio-temporal cues across various scales. However, existing methods often overlook the multi-granularity nature of actions. To address this limitation, we introduce the Context-Aware Network (CAN). CAN consists of two core modules: the Multi-scale Temporal Cue Module (MTCM) and the Group Spatial Cue Module (GSCM). MTCM effectively extracts temporal cues at multiple scales, capturing both fast-changing motion details and overall action flow. GSCM, on the other hand, extracts spatial cues at different scales by grouping feature maps and applying specialized extraction methods to each group. Experiments conducted on five benchmark datasets (Something-Something V1 and V2, Diving48, Kinetics-400, and UCF101) demonstrate the effectiveness of CAN. Our approach achieves competitive performance, outperforming most mainstream methods, with accuracies of 50.4% on Something-Something V1, 63.9% on Something-Something V2, 88.4% on Diving48, 74.9% on Kinetics-400, and 86.9% on UCF101. These results highlight the importance of capturing multi-scale spatio-temporal cues for robust action recognition.

Paper Structure

This paper contains 28 sections, 10 equations, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Spatio-temporal feature learning
  4. Attention mechanisms for action recognition
  5. Skeleton-based action recognition
  6. Methodology
  7. Overview
  8. Multi-scale temporal cue module
  9. Group spatial cue module
  10. Residual connection
  11. Pointwise motion module
  12. Local motion module
  13. Global motion module
  14. Experiments
  15. Datasets
  16. ...and 13 more sections

Figures (4)

  • Figure 1: The overall architecture of CAN is instantiated with the ResNet-50 backbone. CAN consists of two key components: the MTCM, which captures motion cues at multiple temporal scales, and the GSCM, which extracts motion cues at various spatial scales.
  • Figure 2: Illustration of each path of GSCM, including Global Motion Module (GMM), Pointwise Motion Module (PMM), and Local Motion Module (LMM).
  • Figure 3: Performance analysis of our CAN on specific action categories from the Kinetics-400 dataset.
  • Figure 4: Visualization of important features extracted by TSM, GC-TSN, and CAN. Compared to TSM and GC-TSN, CAN can focus on the region where the action occurs in each frame.