Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Representation Learning for Compressed Video Action Recognition via Attentive Cross-modal Interaction with Motion Enhancement

Bing Li, Jiaxin Chen, Dongming Zhang, Xiuguo Bao, Di Huang

TL;DR

The paper tackles action recognition in compressed videos where motion cues are coarse and fusion is suboptimal. It introduces MEACI-Net with a CME network (MSB + DM) to enhance motion, and ACI (SMC + CMA) for cross-modal fusion between RGB I-frames and motion P-frames. On HMDB-51, UCF-101, and Kinetics-400, MEACI-Net achieves state-of-the-art results among compressed-video methods while using substantially fewer GFLOPs and enabling faster inference, thereby narrowing the gap to raw-video approaches. This work provides a practical pathway for efficient video understanding in settings with limited bandwidth and compute resources.

Abstract

Compressed video action recognition has recently drawn growing attention, since it remarkably reduces the storage and computational cost via replacing raw videos by sparsely sampled RGB frames and compressed motion cues (e.g., motion vectors and residuals). However, this task severely suffers from the coarse and noisy dynamics and the insufficient fusion of the heterogeneous RGB and motion modalities. To address the two issues above, this paper proposes a novel framework, namely Attentive Cross-modal Interaction Network with Motion Enhancement (MEACI-Net). It follows the two-stream architecture, i.e. one for the RGB modality and the other for the motion modality. Particularly, the motion stream employs a multi-scale block embedded with a denoising module to enhance representation learning. The interaction between the two streams is then strengthened by introducing the Selective Motion Complement (SMC) and Cross-Modality Augment (CMA) modules, where SMC complements the RGB modality with spatio-temporally attentive local motion features and CMA further combines the two modalities with selective feature augmentation. Extensive experiments on the UCF-101, HMDB-51 and Kinetics-400 benchmarks demonstrate the effectiveness and efficiency of MEACI-Net.

Representation Learning for Compressed Video Action Recognition via Attentive Cross-modal Interaction with Motion Enhancement

TL;DR

The paper tackles action recognition in compressed videos where motion cues are coarse and fusion is suboptimal. It introduces MEACI-Net with a CME network (MSB + DM) to enhance motion, and ACI (SMC + CMA) for cross-modal fusion between RGB I-frames and motion P-frames. On HMDB-51, UCF-101, and Kinetics-400, MEACI-Net achieves state-of-the-art results among compressed-video methods while using substantially fewer GFLOPs and enabling faster inference, thereby narrowing the gap to raw-video approaches. This work provides a practical pathway for efficient video understanding in settings with limited bandwidth and compute resources.

Abstract

Compressed video action recognition has recently drawn growing attention, since it remarkably reduces the storage and computational cost via replacing raw videos by sparsely sampled RGB frames and compressed motion cues (e.g., motion vectors and residuals). However, this task severely suffers from the coarse and noisy dynamics and the insufficient fusion of the heterogeneous RGB and motion modalities. To address the two issues above, this paper proposes a novel framework, namely Attentive Cross-modal Interaction Network with Motion Enhancement (MEACI-Net). It follows the two-stream architecture, i.e. one for the RGB modality and the other for the motion modality. Particularly, the motion stream employs a multi-scale block embedded with a denoising module to enhance representation learning. The interaction between the two streams is then strengthened by introducing the Selective Motion Complement (SMC) and Cross-Modality Augment (CMA) modules, where SMC complements the RGB modality with spatio-temporally attentive local motion features and CMA further combines the two modalities with selective feature augmentation. Extensive experiments on the UCF-101, HMDB-51 and Kinetics-400 benchmarks demonstrate the effectiveness and efficiency of MEACI-Net.
Paper Structure (26 sections, 5 equations, 6 figures, 7 tables)

This paper contains 26 sections, 5 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Action Recognition in Raw Videos
  4. Action Recognition in Compressed Videos
  5. The Proposed Approach
  6. Compressed Video Encapsulation
  7. Compressed Motion Enhancement Network
  8. The Multi-Scale Block (MSB).
  9. The Denoising Module (DM).
  10. Attentive Cross-modal Interaction
  11. The Selective Motion Complement (SMC) Unit.
  12. The Cross-Modality Augment (CMA) Unit.
  13. Experimental Results and Analysis
  14. Datasets and Evaluation Protocol
  15. Implementation Details
  16. ...and 11 more sections

Figures (6)

  • Figure 1: (b) and (c) show coarse and noisy motion vectors compared to optical flows. Images in one row correspond to the same frame. Interferential backgrounds are highlighted by red rectangles.
  • Figure 2: Framework of the proposed MEACI-Net. We encapsulate I-frame, Motion Vector and Residual (MVR) from compressed videos and constitutes the RGB and MVR modalities, respectively. I3D-ResNet50 is directly employed to process I-frame clips while the CME network is designed to work with P-frame clips. The SMC and CMA facilitate cross-modal interaction in feature fusion from two modalities.
  • Figure 3: Network structures: (a) the Bottleneck Block in I3D-ResNet50; (b) the Multi-Scale Block; (c) the Denoising Module.
  • Figure 4: Visualization of feature maps by Grad-CAM on UCF-101.
  • Figure A: Module details of (a) the Selective Motion Complement (SMC) unit and (b) the Cross-Modality Augment (CMA) unit.
  • ...and 1 more figures