Multichannel Orthogonal Transform-Based Perceptron Layers for Efficient ResNets

TL;DR

This work introduces transform-domain perceptron layers that replace select $3\times3$ convolutions in CNNs with orthogonal transform-based operations ($DCT$, $HT$, $BWT$) plus trainable soft-thresholding. By leveraging transform-domain convolution theorems, these layers perform filtering via simple elementwise multiplications and a learnable nonlinear threshold, yielding location- and channel-specific representations with significantly fewer parameters and MACs. Across CIFAR-10/100 and ImageNet-1K, the proposed layers (especially multi-channel DCT/BWT/HT variants) achieve competitive or superior accuracy while reducing model complexity; an extra single-channel transform layer before pooling can provide additional gains. The approach offers a practical path to more efficient ResNets and can be extended to other CNNs, though memory usage and padding requirements for non-power-of-two sizes pose challenges for deployment.

Abstract

In this paper, we propose a set of transform-based neural network layers as an alternative to the $3\times3$ Conv2D layers in Convolutional Neural Networks (CNNs). The proposed layers can be implemented based on orthogonal transforms such as the Discrete Cosine Transform (DCT), Hadamard transform (HT), and biorthogonal Block Wavelet Transform (BWT). Furthermore, by taking advantage of the convolution theorems, convolutional filtering operations are performed in the transform domain using element-wise multiplications. Trainable soft-thresholding layers, that remove noise in the transform domain, bring nonlinearity to the transform domain layers. Compared to the Conv2D layer, which is spatial-agnostic and channel-specific, the proposed layers are location-specific and channel-specific. Moreover, these proposed layers reduce the number of parameters and multiplications significantly while improving the accuracy results of regular ResNets on the ImageNet-1K classification task. Furthermore, they can be inserted with a batch normalization layer before the global average pooling layer in the conventional ResNets as an additional layer to improve classification accuracy.

Figures (5)

• Figure 1: $N=4$ band ($M=2$ stage) subband decomposition filter bank structure for the orthogonal transform. The length of $\mathbf{x}$ is $N$. The length of each $\mathbf{x}_i$ is $\frac{N}{4}$ for $i=0, 1, 2, 3$. $\mathbf{X}=\{\mathbf{x}_0, \mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3\}$ is the $N$-length output of the orthogonal transform. In HT, $\mathbf{h}[n] = \{ 1, 1 \}$ and $\mathbf{g}[n] = \{ -1, 1 \}$. In BWT using Bior 1.3 coefficients, $\mathbf{h}[n] = \{-0.125, 0.125, 1, 1, 0.125, -0.125\}$ and $g[n] = \{-1, 1\}$singh2011jpeg.
• Figure 2: (a) Single-channel perceptron layer and (b) multi-channel perceptron layer. Zero padding and truncation are applied in the HT-perceptron layers and the BWT-perceptron layers if the image size is not a power of 2.
• Figure 3: Processing in the transform domain.
• Figure 4: (a) ResNet's convolutional residual blocks V1 (left) and V2 (right) versus (b) the proposed transform-based Perceptron (TP) residual blocks V1 (left) and V2 (right). BN stands for batch normalization. One $3\times3$ Conv2D layer in each block is replaced by one proposed DCT-perceptron layer.
• Figure 5: Training on CIFAR-10, CIFAR-100, and ImageNet-1K. In (a) and (b), curves denote test error. In (c) and (d), curves denote the validation error of the center crops. With reduced parameters and MACs, comparable CIFAR-10 accuracy results, better CIFAR-100 accuracy results, and better ImageNet-1K accuracy results are obtained using the proposed tri-channel DCT/HT/BWT-perceptron layers.

Theorems & Definitions (2)

• Theorem 1: DCT convolution theorem
• Theorem 2: HT convolution theorem