Sparse Hybrid Linear-Morphological Networks
Konstantinos Fotopoulos, Christos Garoufis, Petros Maragos
TL;DR
The paper tackles neural network efficiency by leveraging pruning-friendly morphological layers within a hybrid linear-morphological architecture. It replaces activations with sparse morphological layers inserted between linear blocks, and evaluates several heads including ReLU, Maxout, dense morphologies, and a sparsely initialized variant. The results show competitive accuracy across MTAT and CIFAR-10, with the sparse morphological design yielding the best pruning performance and faster early convergence, while inducing sparsity in the linear layers themselves. This approach offers practical gains in model compression and deployment efficiency, particularly for edge systems.
Abstract
We investigate hybrid linear-morphological networks. Recent studies highlight the inherent affinity of morphological layers to pruning, but also their difficulty in training. We propose a hybrid network structure, wherein morphological layers are inserted between the linear layers of the network, in place of activation functions. We experiment with the following morphological layers: 1) maxout pooling layers (as a special case of a morphological layer), 2) fully connected dense morphological layers, and 3) a novel, sparsely initialized variant of (2). We conduct experiments on the Magna-Tag-A-Tune (music auto-tagging) and CIFAR-10 (image classification) datasets, replacing the linear classification heads of state-of-the-art convolutional network architectures with our proposed network structure for the various morphological layers. We demonstrate that these networks induce sparsity to their linear layers, making them more prunable under L1 unstructured pruning. We also show that on MTAT our proposed sparsely initialized layer achieves slightly better performance than ReLU, maxout, and densely initialized max-plus layers, and exhibits faster initial convergence.