Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Efficient 3D Convolutional Neural Network with Channel-wise, Spatial-grouped, and Temporal Convolutions

Zhe Wang, Xulei Yang

TL;DR

This paper introduces SEVNets, a simple and efficient 3D CNN architecture for video action recognition built from SEV Modules, which decompose 3D convolution into channel-wise, spatial-grouped, and temporal operations. Trained from scratch on multiple datasets without ImageNet pretraining, SEVNets achieve competitive or superior accuracy at substantially lower FLOPs and parameter counts, including state-of-the-art results on Something-Something V1/V2, MMIT, and FineGym with RGB inputs. The authors also show ablations confirming the benefit of spatial-grouped convolution, demonstrate compatibility with SE attention units, and discuss compression via 8-bit quantization, underscoring practical deployment advantages.

Abstract

There has been huge progress on video action recognition in recent years. However, many works focus on tweaking existing 2D backbones due to the reliance of ImageNet pretraining, which restrains the models from achieving higher efficiency for video recognition. In this work we introduce a simple and very efficient 3D convolutional neural network for video action recognition. The design of the building block consists of a channel-wise convolution, followed by a spatial group convolution, and finally a temporal convolution. We evaluate the performance and efficiency of our proposed network on several video action recognition datasets by directly training on the target dataset without relying on pertaining. On Something-Something-V1&V2, Kinetics-400 and Multi-Moments in Time, our network can match or even surpass the performance of other models which are several times larger. On the fine-grained action recognition dataset FineGym, we beat the previous state-of-the-art accuracy achieved with 2-stream methods by more than 5% using only RGB input.

An Efficient 3D Convolutional Neural Network with Channel-wise, Spatial-grouped, and Temporal Convolutions

TL;DR

This paper introduces SEVNets, a simple and efficient 3D CNN architecture for video action recognition built from SEV Modules, which decompose 3D convolution into channel-wise, spatial-grouped, and temporal operations. Trained from scratch on multiple datasets without ImageNet pretraining, SEVNets achieve competitive or superior accuracy at substantially lower FLOPs and parameter counts, including state-of-the-art results on Something-Something V1/V2, MMIT, and FineGym with RGB inputs. The authors also show ablations confirming the benefit of spatial-grouped convolution, demonstrate compatibility with SE attention units, and discuss compression via 8-bit quantization, underscoring practical deployment advantages.

Abstract

There has been huge progress on video action recognition in recent years. However, many works focus on tweaking existing 2D backbones due to the reliance of ImageNet pretraining, which restrains the models from achieving higher efficiency for video recognition. In this work we introduce a simple and very efficient 3D convolutional neural network for video action recognition. The design of the building block consists of a channel-wise convolution, followed by a spatial group convolution, and finally a temporal convolution. We evaluate the performance and efficiency of our proposed network on several video action recognition datasets by directly training on the target dataset without relying on pertaining. On Something-Something-V1&V2, Kinetics-400 and Multi-Moments in Time, our network can match or even surpass the performance of other models which are several times larger. On the fine-grained action recognition dataset FineGym, we beat the previous state-of-the-art accuracy achieved with 2-stream methods by more than 5% using only RGB input.

Paper Structure

This paper contains 15 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Simple and Efficient Video Networks
  4. SEV Module
  5. SEVNets
  6. Experiments
  7. Training and Testing
  8. Experiments: Something-Something V1 & V2
  9. Experiments: Multi-Moments in Time
  10. Experiments: Kinetics-400
  11. Experiments: Fine-grained Action Recognition on FineGym
  12. Experiments: Ablation study
  13. Experiments: Compatibility
  14. Discussions: Complexity
  15. Conclusion

Figures (4)

  • Figure 1: Accuracy against complexity (FLOPs) plot of state-of-the-arts models under 35 GFLOPs on Something-Something-V2.
  • Figure 2: Structure of SEV Module
  • Figure 3: Accuracy against number of parameters plot of state-of-the-arts models under 35 GFLOPs on Something-Something-V2.
  • Figure 4: "R(2+1)D" and "R3D" like structures for ablation. (Numbers in brackets indicate number of output channels)