Symmetry Breaking in Neural Network Optimization: Insights from Input Dimension Expansion

TL;DR

A metric to quantify symmetry breaking in networks is developed, revealing its role in common optimization methods and its connection to properties like equivariance and providing actionable insights for improving model efficiency.

Abstract

Understanding the mechanisms behind neural network optimization is crucial for improving network design and performance. While various optimization techniques have been developed, a comprehensive understanding of the underlying principles that govern these techniques remains elusive. Specifically, the role of symmetry breaking, a fundamental concept in physics, has not been fully explored in neural network optimization. This gap in knowledge limits our ability to design networks that are both efficient and effective. Here, we propose the symmetry breaking hypothesis to elucidate the significance of symmetry breaking in enhancing neural network optimization. We demonstrate that a simple input expansion can significantly improve network performance across various tasks, and we show that this improvement can be attributed to the underlying symmetry breaking mechanism. We further develop a metric to quantify the degree of symmetry breaking in neural networks, providing a practical approach to evaluate and guide network design. Our findings confirm that symmetry breaking is a fundamental principle that underpins various optimization techniques, including dropout, batch normalization, and equivariance. By quantifying the degree of symmetry breaking, our work offers a practical technique for performance enhancement and a metric to guide network design without the need for complete datasets and extensive training processes.

Figures (10)

• Figure 1: Performance comparison for raw data and expanded data on CIFAR and ImageNet datasets with various models. Each row represents a different dataset: CIFAR-10, CIFAR-100, ImageNet-R, and ImageNet-100, respectively. Each column corresponds to a different model architecture: ResNet-18, ResNet-50, DenseNet-121, MobileNet-v3, and EfficientNet. Within each cell, the plot shows two curves representing test accuracy over epochs: the red curve for expanded data and the blue curve for raw data.
• Figure 2: Performance comparison for raw data and expanded data on various datasets using ResNet-18. Each bar represents the test accuracy, with blue bars indicating the accuracy for raw data and red bars showing the additional accuracy gained from using expanded data. The total length of each bar (blue plus red) represents the final test accuracy after applying dimension expansion. Up-arrows next to each dataset indicate the accuracy increment achieved through dimension expansion, with the numerical values beside the arrows quantifying the improvement.
• Figure 3: Comparison of the predicted QCD state equation results with expanded (black scatter) and unexpanded (red scatter) input dimensions. (A) and (B) show the mean absolute error of the normalized pressure ($P/T^4$) and energy density ($E/T^4$) as functions of the temperature. The critical temperature $T_c = 0.155$ GeV.
• Figure 4: Comparison of loss distributions for 15 PDEs using raw and expanded input dimensions. Each subfigure represents the results for 5 different PDEs. The loss values have been normalized using the maximum loss for each PDE to facilitate comparison. The blue distribution on the left side of each violin plot represents the raw input dimensions, while the light red distribution on the right side represents the expanded input dimensions. The plots show the distribution of loss values across different random seeds, with the expanded input dimensions generally resulting in lower loss values, indicating improved performance.
• Figure 5: Ising model. (A) Schematic diagram of the Ising model composed of a two-dimensional periodic lattice, typically arranged in a square grid. Each lattice point represents a spin variable, where spin-up is denoted by +1 and spin-down by -1. (B) The energy landscape of the Ising model. The two curves represent the energy landscape for $h=0$ and $h=0.45$. When $h=0.45$, the symmetry of the system is broken.