Connectome-Guided Automatic Learning Rates for Deep Networks

TL;DR

CG-ALR addresses the problem that conventional learning-rate schedules do not adapt to the evolving internal representations of deep networks. It builds a functional connectome from neuron activations and quantifies its reconfiguration with persistent-homology distances such as $TOP$ and $WD$, using a median–MAD robust signal $z_t$ to gate LR updates with hysteresis while preserving a Robbins–Monro envelope. The authors provide convergence guarantees under mild assumptions and validate the method across image and graph benchmarks, showing competitive or superior performance to SGD-based schedules and parameter-free methods like DoG. This brain-inspired approach offers a principled, interpretable mechanism to accelerate training when representations stabilize and to stabilize training during reconfiguration, with broad applicability to vision and graph learning tasks.

Abstract

The human brain is highly adaptive: its functional connectivity reconfigures on multiple timescales during cognition and learning, enabling flexible information processing. By contrast, artificial neural networks typically rely on manually-tuned learning-rate schedules or generic adaptive optimizers whose hyperparameters remain largely agnostic to a model's internal dynamics. In this paper, we propose Connectome-Guided Automatic Learning Rate (CG-ALR) that dynamically constructs a functional connectome of the neural network from neuron co-activations at each training iteration and adjusts learning rates online as this connectome reconfigures. This connectomics-inspired mechanism adapts step sizes to the network's dynamic functional organization, slowing learning during unstable reconfiguration and accelerating it when stable organization emerges. Our results demonstrate that principles inspired by brain connectomes can inform the design of adaptive learning rates in deep learning, generally outperforming traditional SGD-based schedules and recent methods.

Figures (3)

• Figure 1: Learning Rate Dynamics Across Epochs on CIFAR-10. Comparison of our proposed CG-ALR with TOP against standard schedules that require initial learning rates: SGD-Cosine, SGD-Step, SGD-Exp, and SGD-Plateau.
• Figure 2: (a) Training Loss on Three Datasets (CIFAR-100, ENZYMES, PROTEINS). Solid lines are the averages across seeds for the best learning-rate choice from the predefined grid ($\eta^\star \!\in\!\{0.1,0.01,0.001\}$ for CG-ALR and SGD-based schedules; $\eta\!\in\!\{0.1,0.01,0.001\}$ for SGD-Constant; default hyperparameters for DoG). Dashed lines are the averages across seeds for the worst choice from the same grid. Shaded bands denote the min–max range across three seeds. For CG-ALR, we use TOP on CIFAR-100 and SWK on ENZYMES/PROTEINS. (b) Validation Loss on the Same Three Datasets for the Same Set of Optimizers.
• Figure 3: (a) Training Loss on three datasets (CIFAR-10, Mini-ImageNet, and MUTAG). (b) Validation Loss on the same datasets. Solid lines indicate the averages across three seeds using the best learning-rate configuration from the predefined grid ($\eta^\star\!\in\!\{0.1,0.01,0.001\}$ for CG-ALR and SGD-based schedules; $\eta\!\in\!\{0.1,0.01,0.001\}$ for SGD-Constant; and default hyperparameters for DoG). Dashed lines denote the corresponding worst configurations. Shaded regions represent the min–max range across seeds. All methods share the same model backbone and training protocol.

Theorems & Definitions (12)

• Theorem 1
• proof
• Remark 1
• Lemma 1: Robbins--Monro conditions preserved
• proof
• Lemma 2: One-step expected descent
• proof
• Lemma 3: Robbins--Siegmund, almost supermartingale
• Theorem 2: Convergence to stationary points
• proof