Impact of Batch Normalization on Convolutional Network Representations

TL;DR

Batch Normalization accelerates training and enhances generalization, but the mechanisms are debated. This work probes BN's impact on CNN internal representations, focusing on representational sparsity and clustering across MNIST and CIFAR-10 architectures. It finds that sparsity does not reliably predict generalization, while BN tends to induce purer, more class-consistent and early-class-aligned clusters, suggesting clustering structure as a plausible mechanism for BN-driven gains. The results motivate representation-level diagnostics and broader investigations across diverse datasets and models.

Abstract

Batch normalization (BatchNorm) is a popular layer normalization technique used when training deep neural networks. It has been shown to enhance the training speed and accuracy of deep learning models. However, the mechanics by which BatchNorm achieves these benefits is an active area of research, and different perspectives have been proposed. In this paper, we investigate the effect of BatchNorm on the resulting hidden representations, that is, the vectors of activation values formed as samples are processed at each hidden layer. Specifically, we consider the sparsity of these representations, as well as their implicit clustering -- the creation of groups of representations that are similar to some extent. We contrast image classification models trained with and without batch normalization and highlight consistent differences observed. These findings highlight that BatchNorm's effect on representational sparsity is not a significant factor affecting generalization, while the representations of models trained with BatchNorm tend to show more advantageous clustering characteristics.

Figures (6)

• Figure 1: Average channel sparsity for BatchNorm (blue) and Non-BatchNorm (orange) models for selected convolutional layers of the standard CNN models trained on MNIST. Error bars show the standard deviation across $4$ random initialization seeds.
• Figure 2: Average channel sparsity for BatchNorm (blue) and Non-BatchNorm (orange) models in selected VGG-16 convolutional layers trained on CIFAR10, with error bars showing the standard deviation across $4$ random initialization seeds.
• Figure 3: Average layer sparsity for models trained with (blue) and without (orange) BatchNorm. Error bars indicate the standard deviation across $4$ random initialization seeds.
• Figure 4: Average class-based DBI cluster purity score per layer for BatchNorm (blue) and Non-BatchNorm (orange) models, with error bars showing standard deviation across $4$ random initialization seeds. Lower scores indicate higher purity.
• Figure 5: Average optimal number of class-agnostic clusters per layer for BatchNorm (blue) and Non-BatchNorm (orange) models, with error bars showing the standard deviation across $4$ random initialization seeds.