Normalized Convolutional Neural Network

TL;DR

This work introduces Normalized Convolution (NC), a convolution-centric normalization that standardizes the rows of the im2col matrix during the convolution operation, enabling adaptive handling of sliced inputs and kernel structures. The authors provide theoretical arguments that NC reduces Lipschitz constants of the loss and gradients, yielding a smoother optimization landscape, and substantiate these claims with extensive experiments across ImageNet, CIFAR, COCO, PASCAL VOC, and generative modeling tasks. Empirically, NC consistently outperforms Group Normalization in micro-batch settings and provides gains when combined with BN or GN, with notable improvements in stability and training speed. The work also demonstrates a CUDA-accelerated implementation to address computational efficiency and discusses relationships with Positional Normalization, suggesting NC as a general, universal normalization mechanism for convolutional networks with potential broad impact on practice.

Abstract

We introduce a Normalized Convolutional Neural Layer, a novel approach to normalization in convolutional networks. Unlike conventional methods, this layer normalizes the rows of the im2col matrix during convolution, making it inherently adaptive to sliced inputs and better aligned with kernel structures. This distinctive approach differentiates it from standard normalization techniques and prevents direct integration into existing deep learning frameworks optimized for traditional convolution operations. Our method has a universal property, making it applicable to any deep learning task involving convolutional layers. By inherently normalizing within the convolution process, it serves as a convolutional adaptation of Self-Normalizing Networks, maintaining their core principles without requiring additional normalization layers. Notably, in micro-batch training scenarios, it consistently outperforms other batch-independent normalization methods. This performance boost arises from standardizing the rows of the im2col matrix, which theoretically leads to a smoother loss gradient and improved training stability.

Figures (4)

• Figure 1: Normalized convolutional layer diagram. The rows of im2col matrix in the pink area are stadardized and the correspond area of weights can be also stadardized.
• Figure 2: The graph illustrates the train and test loss values during training and validation on CIFAR-10. It can be observed that the training process is more stable, and the loss value decreases at a faster rate.
• Figure 3: A pair of test images is presented. The first two images are the original images, the third image is generated using PONO, and the last image is produced using NC.
• Figure 4: The images were obtained using CycleGAN. The first image represents the original, the second image is generated using PONO, and the third image is produced using NC. Notably, the image obtained from NC contains more detailed information, which can be intuitively observed.