Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?

Habib Hajimolahoseini, Walid Ahmed, Austin Wen, Yang Liu

TL;DR

This paper questions the necessity of 3D convolutions operating on 5D video tensors ($[B, T, X, Y, C]$) and presents a framework to simulate 3D spatio-temporal processing using 4D inputs and 3D/2D kernels. It introduces a dual-branch architecture with Spatial Analysis (2D convolutions) and Temporal Analysis (dimension-wise operations) that operate on reshaped tensors and fuse to form a 4D-compatible representation, achieving outputs of the form $[B, \frac{T}{s}, \frac{X}{s}\times\frac{Y}{s}, S]$. Empirical results on ECO-Lite show notable improvements in efficiency (fewer parameters and FLOPs) and accuracy across benchmarks like Kinetics-400 and UCF-101, with the Cat variant often delivering the best accuracy and the Add variant offering the best speed. The approach demonstrates strong potential for real-time video analysis on edge devices and accelerators, reducing memory usage while maintaining or enhancing performance.

Abstract

In this paper, we present a comprehensive study and propose several novel techniques for implementing 3D convolutional blocks using 2D and/or 1D convolutions with only 4D and/or 3D tensors. Our motivation is that 3D convolutions with 5D tensors are computationally very expensive and they may not be supported by some of the edge devices used in real-time applications such as robots. The existing approaches mitigate this by splitting the 3D kernels into spatial and temporal domains, but they still use 3D convolutions with 5D tensors in their implementations. We resolve this issue by introducing some appropriate 4D/3D tensor reshaping as well as new combination techniques for spatial and temporal splits. The proposed implementation methods show significant improvement both in terms of efficiency and accuracy. The experimental results confirm that the proposed spatio-temporal processing structure outperforms the original model in terms of speed and accuracy using only 4D tensors with fewer parameters.

Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?

TL;DR

This paper questions the necessity of 3D convolutions operating on 5D video tensors () and presents a framework to simulate 3D spatio-temporal processing using 4D inputs and 3D/2D kernels. It introduces a dual-branch architecture with Spatial Analysis (2D convolutions) and Temporal Analysis (dimension-wise operations) that operate on reshaped tensors and fuse to form a 4D-compatible representation, achieving outputs of the form . Empirical results on ECO-Lite show notable improvements in efficiency (fewer parameters and FLOPs) and accuracy across benchmarks like Kinetics-400 and UCF-101, with the Cat variant often delivering the best accuracy and the Add variant offering the best speed. The approach demonstrates strong potential for real-time video analysis on edge devices and accelerators, reducing memory usage while maintaining or enhancing performance.

Abstract

In this paper, we present a comprehensive study and propose several novel techniques for implementing 3D convolutional blocks using 2D and/or 1D convolutions with only 4D and/or 3D tensors. Our motivation is that 3D convolutions with 5D tensors are computationally very expensive and they may not be supported by some of the edge devices used in real-time applications such as robots. The existing approaches mitigate this by splitting the 3D kernels into spatial and temporal domains, but they still use 3D convolutions with 5D tensors in their implementations. We resolve this issue by introducing some appropriate 4D/3D tensor reshaping as well as new combination techniques for spatial and temporal splits. The proposed implementation methods show significant improvement both in terms of efficiency and accuracy. The experimental results confirm that the proposed spatio-temporal processing structure outperforms the original model in terms of speed and accuracy using only 4D tensors with fewer parameters.
Paper Structure (6 sections, 4 figures, 4 tables)

This paper contains 6 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Spatial Analysis
  4. Temporal Analysis
  5. Experimental Results
  6. Conclusion

Figures (4)

  • Figure 1: A regular 3D convolutional layer applied to 5D tensors.
  • Figure 2: Reshaping the input 5D tensor into 4D.
  • Figure 3: The proposed architecture which replaces the regular 3D convolutional layers.
  • Figure 4: Number of parameters vs. speed and accuracy improvement for different methods. Note that the lower the number of parameters and the higher the speed and accuracy, the better the model.