Layer-Specific Optimization: Sensitivity Based Convolution Layers Basis Search

TL;DR

This paper tackles the problem of overparameterization in CNNs by introducing basis convolutions: each Conv2d layer trains a small set of basis filters $C_{out}'$, while remaining outputs are reconstructed as linear combinations via a trainable coefficient matrix. It develops a layer-wise sensitivity analysis to identify which layers can tolerate basis convolutions with minimal impact on final accuracy, enabling selective deployment that reduces model size and can accelerate training. The approach includes forward-cost and backward-cost analyses, methods to shrink the coefficient matrix, and a practical skip-training scheme. Empirical results on ResNet18 trained on CIFAR-10 show that selective basis convolutions can yield training-time reductions and smaller models with limited accuracy loss, whereas naive full-layer deployment can degrade performance. The work points to a scalable path for parameter-efficient training and offers a framework for automatic layer-subset selection to balance speed and accuracy in practice.

Abstract

Deep neural network models have a complex architecture and are overparameterized. The number of parameters is more than the whole dataset, which is highly resource-consuming. This complicates their application and limits its usage on different devices. Reduction in the number of network parameters helps to reduce the size of the model, but at the same time, thoughtlessly applied, can lead to a deterioration in the quality of the network. One way to reduce the number of model parameters is matrix decomposition, where a matrix is represented as a product of smaller matrices. In this paper, we propose a new way of applying the matrix decomposition with respect to the weights of convolutional layers. The essence of the method is to train not all convolutions, but only the subset of convolutions (basis convolutions), and represent the rest as linear combinations of the basis ones. Experiments on models from the ResNet family and the CIFAR-10 dataset demonstrate that basis convolutions can not only reduce the size of the model but also accelerate the forward and backward passes of the network. Another contribution of this work is that we propose a fast method for selecting a subset of network layers in which the use of matrix decomposition does not degrade the quality of the final model.

Figures (11)

• Figure 1: Matrix decomposition.
• Figure 2: Basis convolutions for the forward's output modification.
• Figure 3: Dependencies of one output scalar on input scalars for Conv2d layer. Forward method's output modification with basis convs speeds up forward pass but slows down backward pass.
• Figure 4: Some basis convolutions' output stays unchanged as the subset of the layer's resulting output.
• Figure 5: Validation accuracy for models where basis convolutions are applied in different layer groups ("light" and "heavy").