Deep residual learning with product units

TL;DR

This work introduces PURe, a product‑unit residual network that inserts 2D product units into the second layer of each residual block to enable multiplicative feature interactions while removing ReLU activations to preserve information. The core math leverages y = ∏_{i=1}^{n} x_i^{w_i} with a log–exp reformulation y = exp(∑ w_i log x_i) for stable computation, extended to 2D kernels as y(i,j) = ∏_{m,n} x(i+m, j+n)^{w(m,n)}. Across Galaxy10 DECaLS, ImageNet, and CIFAR‑10, PURe variants—especially PURe34—achieve competitive or superior accuracy with far fewer parameters and faster convergence than ResNets, and exhibit improved robustness to noise. The results demonstrate that multiplicative feature interactions in a residual framework can yield strong generalization and efficiency, with the potential for extensions to complex‑valued units and other vision tasks like detection and segmentation.

Abstract

We propose a deep product-unit residual neural network (PURe) that integrates product units into residual blocks to improve the expressiveness and parameter efficiency of deep convolutional networks. Unlike standard summation neurons, product units enable multiplicative feature interactions, potentially offering a more powerful representation of complex patterns. PURe replaces conventional convolutional layers with 2D product units in the second layer of each residual block, eliminating nonlinear activation functions to preserve structural information. We validate PURe on three benchmark datasets. On Galaxy10 DECaLS, PURe34 achieves the highest test accuracy of 84.89%, surpassing the much deeper ResNet152, while converging nearly five times faster and demonstrating strong robustness to Poisson noise. On ImageNet, PURe architectures outperform standard ResNet models at similar depths, with PURe34 achieving a top-1 accuracy of 80.27% and top-5 accuracy of 95.78%, surpassing deeper ResNet variants (ResNet50, ResNet101) while utilizing significantly fewer parameters and computational resources. On CIFAR-10, PURe consistently outperforms ResNet variants across varying depths, with PURe272 reaching 95.01% test accuracy, comparable to ResNet1001 but at less than half the model size. These results demonstrate that PURe achieves a favorable balance between accuracy, efficiency, and robustness. Compared to traditional residual networks, PURe not only achieves competitive classification performance with faster convergence and fewer parameters, but also demonstrates greater robustness to noise. Its effectiveness across diverse datasets highlights the potential of product-unit-based architectures for scalable and reliable deep learning in computer vision.

Figures (5)

• Figure 1: Computational flow of the 2D product unit with a kernel of size $3 \times 3$. $\theta$ is a trainable threshold parameter.
• Figure 2: Comparison between standard and product-unit residual blocks. $\text{Conv}^d$ denotes a convolutional layer used for dimension matching, and ConvPU represents a 2D product unit.
• Figure 3: Architecture of the proposed product-unit residual network.
• Figure 4: Example images from each of the ten classes in the Galaxy10 DECaLS dataset.
• Figure 5: Validation loss and accuracy curves during training on Galaxy10 DECaLS. All curves are smoothed for visual clarity using a moving average. Results reflect the average performance across five independent runs.