The Early Phase of Neural Network Training

TL;DR

The paper investigates the very early phase of deep network training using Iterative Magnitude Pruning with Rewinding to quantify how weights, signs, and sub-networks evolve. It shows that the early period comprises distinct sub-phases with rapid changes, and it reveals that deep networks are not robust to random-weight reinitialization even when signs are preserved, while weight magnitudes and non-i.i.d. distributions play crucial roles. Data pretraining strategies, including self-supervised rotation and input blurring, can approximate early-state transitions, suggesting that labels accelerate but are not strictly required for the initial learning dynamics. These findings have implications for the lottery ticket hypothesis and indicate that appropriate pretraining can substitute for late rewinding in some cases, though network architecture and data properties critically modulate these effects.

Abstract

Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge (Frankle et al., 2019), gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the network undergoes a critical period (Achille et al., 2019). Here, we examine the changes that deep neural networks undergo during this early phase of training. We perform extensive measurements of the network state during these early iterations of training and leverage the framework of Frankle et al. (2019) to quantitatively probe the weight distribution and its reliance on various aspects of the dataset. We find that, within this framework, deep networks are not robust to reinitializing with random weights while maintaining signs, and that weight distributions are highly non-independent even after only a few hundred iterations. Despite this behavior, pre-training with blurred inputs or an auxiliary self-supervised task can approximate the changes in supervised networks, suggesting that these changes are not inherently label-dependent, though labels significantly accelerate this process. Together, these results help to elucidate the network changes occurring during this pivotal initial period of learning.

Figures (22)

• Figure 1: Accuracy of IMP when rewinding to various iterations of the early phase for ResNet-20 sub-networks as a function of sparsity level.
• Figure 2: Rough timeline of the early phase of training for ResNet-20 on CIFAR-10.
• Figure 3: Basic telemetry about the state of ResNet-20 during the first 4000 iterations (10 epochs). Top row: evaluation accuracy/loss; average weight magnitude; percentage of weights that change sign from initialization; the values of ten randomly-selected weights. Bottom row: gradient magnitude; L2 distance of weights from their initial values and final values at the end of training; cosine similarity of weights from their initial values and final values at the end of training.
• Figure 4: Performance of an IMP-derived sub-network of ResNet-20 on CIFAR-10 initialized to the signs at iteration 0 or $k$ and the magnitudes at iteration 0 or $k$. Left: $k=500$. Right: $k=2000$.
• Figure 5: Performance of an IMP-derived ResNet-20 sub-network on CIFAR-10 initialized with the weights at iteration $k$ permuted within various structural elements. Left: $k=500$. Right: $k=2000$.