3DPyranet Features Fusion for Spatio-temporal Feature Learning
Ihsan Ullah, Alfredo Petrosino
TL;DR
The paper introduces a 3D pyramidal neural network (3DPyraNet) designed to learn spatio-temporal features from video data with a biologically inspired pyramid structure and a novel weighting scheme that enables temporal integration while maintaining a compact parameter count. It extends PyraNet with 3D correlation and pooling layers, plus a training framework that supports efficient backpropagation through pyramidal and pooling layers. To leverage learned features, the authors propose two fusion variants, 3DPyraNet-F (global/early fusion) and 3DPyraNet-F_M (local fusion), which feed fused features into a linear SVM classifier, achieving strong performance on action recognition and dynamic scene datasets, often with camera motion. Across Weizmann, KTH, YUPENN, and Maryland datasets, the fusion models outperform several handcrafted descriptors and achieve competitive or state-of-the-art results while significantly reducing model size (about 0.83M parameters vs ~17.5M for C3D). The approach demonstrates robust spatio-temporal learning with lower computational cost, making it suitable for embedded systems and real-world video analysis tasks, and opens avenues for broader evaluation on larger datasets and further interpretability studies.
Abstract
Convolutional neural network (CNN) slides a kernel over the whole image to produce an output map. This kernel scheme reduces the number of parameters with respect to a fully connected neural network (NN). While CNN has proven to be an effective model in recognition of handwritten characters and traffic signal sign boards, etc. recently, its deep variants have proven to be effective in similar as well as more challenging applications like object, scene and action recognition. Deep CNN add more layers and kernels to the classical CNN, increasing the number of parameters, and partly reducing the main advantage of CNN which is less parameters. In this paper, a 3D pyramidal neural network called 3DPyraNet and a discriminative approach for spatio-temporal feature learning based on it, called 3DPyraNet-F, are proposed. 3DPyraNet introduces a new weighting scheme which learns features from both spatial and temporal dimensions analyzing multiple adjacent frames and keeping a biological plausible structure. It keeps the spatial topology of the input image and presents fewer parameters and lower computational and memory costs compared to both fully connected NNs and recent deep CNNs. 3DPyraNet-F extract the features maps of the highest layer of the learned network, fuse them in a single vector, and provide it as input in such a way to a linear-SVM classifier that enhances the recognition of human actions and dynamic scenes from the videos. Encouraging results are reported with 3DPyraNet in real-world environments, especially in the presence of camera induced motion. Further, 3DPyraNet-F clearly outperforms the state-of-the-art on three benchmark datasets and shows comparable result for the fourth.