Neuronal Fluctuations: Learning Rates vs Participating Neurons

TL;DR

This work tackles how the learning rate $\eta$ shapes intrinsic neuronal fluctuations and final performance in deep nets. It adopts a controlled PyTorch autoencoder trained on a synthetic eight-shape dataset, with internal-state hooks capturing weights, biases, activations, and gradients across $\eta \in\{0.01,0.001,0.0001\}$ and 1000 epochs, analyzed via a 'spread of the spread' metric. The findings show that $\eta=0.01$ yields the best reconstruction but many neurons remain largely inactive, while $\eta=0.0001$ engages more neurons yet delivers poorer reconstruction within the fixed training window, illustrating a trade-off between exploration and consolidation. The study provides mechanistic insights into how learning-rate schedules modulate fluctuation dynamics, informing hyperparameter tuning and potentially guiding architecture- and training-schedule design for efficient learning.

Abstract

Deep Neural Networks (DNNs) rely on inherent fluctuations in their internal parameters (weights and biases) to effectively navigate the complex optimization landscape and achieve robust performance. While these fluctuations are recognized as crucial for escaping local minima and improving generalization, their precise relationship with fundamental hyperparameters remains underexplored. A significant knowledge gap exists concerning how the learning rate, a critical parameter governing the training process, directly influences the dynamics of these neural fluctuations. This study systematically investigates the impact of varying learning rates on the magnitude and character of weight and bias fluctuations within a neural network. We trained a model using distinct learning rates and analyzed the corresponding parameter fluctuations in conjunction with the network's final accuracy. Our findings aim to establish a clear link between the learning rate's value, the resulting fluctuation patterns, and overall model performance. By doing so, we provide deeper insights into the optimization process, shedding light on how the learning rate mediates the crucial exploration-exploitation trade-off during training. This work contributes to a more nuanced understanding of hyperparameter tuning and the underlying mechanics of deep learning.

Figures (64)

• Figure 1: Reconstruction with learning rate 0.01
• Figure 2: Reconstruction with learning rate 0.001
• Figure 3: Reconstruction with learning rate 0.0001
• Figure 4: Fluctuations in the weights of the model
• Figure 5: Fluctuations in the biases of the model