A Dimensionality Reduction Approach for Convolutional Neural Networks
Laura Meneghetti, Nicola Demo, Gianluigi Rozza
TL;DR
This work targets deploying convolutional neural networks in memory-constrained environments by introducing a generic reduction framework that splits a pre-trained CNN into a pre-model and a post-model and replaces the post-model with a low-dimensional surrogate. Dimensionality reduction via Active Subspaces or POD is combined with input–output mappings such as Polynomial Chaos Expansion or a small Feedforward Neural Network, enabling accurate replication of the original network’s behavior with far fewer parameters. The approach is trained with knowledge distillation from the full network and demonstrated on VGG-16 with CIFAR-10 and a custom dataset, achieving similar or better accuracy while reducing the checkpoint size from about 56.14 Mb to under 10 Mb; POD+FNN also offers notable training-time savings. The study highlights a trade-off between accuracy and compression controlled by the cut-off layer, and suggests extensions to enable reduction during training for further speedups in optimization, broadening the practical impact for embedded systems.
Abstract
The focus of this paper is the application of classical model order reduction techniques, such as Active Subspaces and Proper Orthogonal Decomposition, to Deep Neural Networks. We propose a generic methodology to reduce the number of layers of a pre-trained network by combining the aforementioned techniques for dimensionality reduction with input-output mappings, such as Polynomial Chaos Expansion and Feedforward Neural Networks. The necessity of compressing the architecture of an existing Convolutional Neural Network is motivated by its application in embedded systems with specific storage constraints. Our experiment shows that the reduced nets obtained can achieve a level of accuracy similar to the original Convolutional Neural Network under examination, while saving in memory allocation.