Enhancing Neural Network Representations with Prior Knowledge-Based Normalization

TL;DR

This paper introduces a new approach to multi-mode normalization that leverages prior knowledge to improve neural network representations and introduces Adaptive Context Normalization (ACN), which dynamically builds contexts in the latent space during training.

Abstract

Deep learning models face persistent challenges in training, particularly due to internal covariate shift and label shift. While single-mode normalization methods like Batch Normalization partially address these issues, they are constrained by batch size dependencies and limiting distributional assumptions. Multi-mode normalization techniques mitigate these limitations but struggle with computational demands when handling diverse Gaussian distributions. In this paper, we introduce a new approach to multi-mode normalization that leverages prior knowledge to improve neural network representations. Our method organizes data into predefined structures, or "contexts", prior to training and normalizes based on these contexts, with two variants: Context Normalization (CN) and Context Normalization - Extended (CN-X). When contexts are unavailable, we introduce Adaptive Context Normalization (ACN), which dynamically builds contexts in the latent space during training. Across tasks in image classification, domain adaptation, and image generation, our methods demonstrate superior convergence and performance.

Figures (4)

• Figure 1: DenseNet-40
• Figure 4: Contrasting Training and Validation Error Curves in CIFAR-100 dataset when using ViT architecture.
• Figure 5: ACN integrated as a normalization layer in a DCGAN. Our results show that incorporating ACN into the DCGAN generator leads to improved (lower) Fréchet Inception Distance (FID) scores.
• Figure 6: Examples of generated images at epoch $200$ are showcased for BN, MixNorm, and ACN in Figure \ref{['fig:bn_gan']},\ref{['fig:mn_gan']}, and\ref{['fig:cn_gan']}, respectively.