Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Two-Time-Scale Learning Dynamics: A Population View of Neural Network Training

Giacomo Borghi, Hyesung Im, Lorenzo Pareschi

Abstract

Population-based learning paradigms, including evolutionary strategies, Population-Based Training (PBT), and recent model-merging methods, combine fast within-model optimisation with slower population-level adaptation. Despite their empirical success, a general mathematical description of the resulting collective training dynamics remains incomplete. We introduce a theoretical framework for neural network training based on two-time-scale population dynamics. We model a population of neural networks as an interacting agent system in which network parameters evolve through fast noisy gradient updates of SGD/Langevin type, while hyperparameters evolve through slower selection--mutation dynamics. We prove the large-population limit for the joint distribution of parameters and hyperparameters and, under strong time-scale separation, derive a selection--mutation equation for the hyperparameter density. For each fixed hyperparameter, the fast parameter dynamics relaxes to a Boltzmann--Gibbs measure, inducing an effective fitness for the slow evolution. The averaged dynamics connects population-based learning with bilevel optimisation and classical replicator--mutator models, yields conditions under which the population mean moves toward the fittest hyperparameter, and clarifies the role of noise and diversity in balancing optimisation and exploration. Numerical experiments illustrate both the large-population regime and the reduced two-time-scale dynamics, and indicate that access to the effective fitness, either in closed form or through population-level estimation, can improve population-level updates.

Two-Time-Scale Learning Dynamics: A Population View of Neural Network Training

Abstract

Population-based learning paradigms, including evolutionary strategies, Population-Based Training (PBT), and recent model-merging methods, combine fast within-model optimisation with slower population-level adaptation. Despite their empirical success, a general mathematical description of the resulting collective training dynamics remains incomplete. We introduce a theoretical framework for neural network training based on two-time-scale population dynamics. We model a population of neural networks as an interacting agent system in which network parameters evolve through fast noisy gradient updates of SGD/Langevin type, while hyperparameters evolve through slower selection--mutation dynamics. We prove the large-population limit for the joint distribution of parameters and hyperparameters and, under strong time-scale separation, derive a selection--mutation equation for the hyperparameter density. For each fixed hyperparameter, the fast parameter dynamics relaxes to a Boltzmann--Gibbs measure, inducing an effective fitness for the slow evolution. The averaged dynamics connects population-based learning with bilevel optimisation and classical replicator--mutator models, yields conditions under which the population mean moves toward the fittest hyperparameter, and clarifies the role of noise and diversity in balancing optimisation and exploration. Numerical experiments illustrate both the large-population regime and the reduced two-time-scale dynamics, and indicate that access to the effective fitness, either in closed form or through population-level estimation, can improve population-level updates.
Paper Structure (50 sections, 7 theorems, 116 equations, 7 figures, 2 algorithms)

This paper contains 50 sections, 7 theorems, 116 equations, 7 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Mathematical modelling of population-based training
  3. The collective training mechanism
  4. Parameters training.
  5. Hyperparameters evolution.
  6. A semi-discrete time evolution.
  7. Description in the many-agent limit
  8. Training operator.
  9. Evolutionary operator.
  10. A PDE model.
  11. Propagation of chaos and convergence
  12. Modelling extensions
  13. Substitution of NNs performing poorly.
  14. Rank-based selection.
  15. Two-time-scale limit
  16. ...and 35 more sections

Key Result

Theorem 4

Let Assumption asm:chaos hold, $t_{\max}>0$ be a computational time horizon, and $f_0\in \mathcal{P}_q(\mathbb{R}^{d_\theta}\times \mathbb{R}^{d_h}), q>1$ an initial distribution. Consider the multi-agent system generated by Algorithm alg:pbt$\{(\theta^i(t),h^i(t))_{t\in [0, t_{\max}]}\}_{i=1}^N$ wi for all $t \in [0, t_{\max}]$.

Figures (7)

  • Figure 1: Two-scale modeling pipeline. Large-population limits ($N\to\infty$) connect the agent-based neural-network dynamics to PDE descriptions; a subsequent two-scale separation yields reduced hyperparameter dynamics and an averaged PDE.
  • Figure 2: Quadratic objective fitness and loss. Median and $10\%-90\%$ quantiles of the population fitness $\mathcal{F}(\theta)$ and loss $\mathcal{L}(\theta,h)$ over $100$ iterations of Algorithm \ref{['alg:pbt']}. PBT updates, where agents are resampled according to their fitness values, occur every 50 training steps and lead to a discontinuity in the fitness evolution. Population size is $N=100$.
  • Figure 3: Many-agent limit$N\to \infty$. Evolution of the hyperparameter distributions for different population sizes $N=10^3,10^4,10^5$ and quadratic problem. Agents are initially sampled uniformly from $[-1,1]$ and evolved according with Algorithm \ref{['alg:pbt']}. The total number of iterations is $500$, with selection jumps at every 50 training steps. The curves are constructed by connecting bin centres (markers) with straight lines and the number of bins is automatically determined.
  • Figure 4: Fast training limit $\varepsilon \to 0$. Evolution of the hyperparameter distributions for different fast-training regimes. The limit $\varepsilon \to 0$ in the PDE model (\ref{['eq:fpde']}) corresponds in Algorithm \ref{['alg:pbt']} to increasing the number of internal SGD steps performed between successive PBT hyperparameter updates. We consider 20, 50, and 100 inner training steps. The case $\varepsilon = 0$ corresponds to directly sampling from the equilibrium distribution (\ref{['eq:toy-steady']}) and evolving the agents according to Algorithm \ref{['alg:macro']}, which simulates the averaged PDE model (\ref{['eq:rhopde']}). The population size is $N=10^5$, and the curves are obtained by connecting bin centers with straight lines using 50 bins.
  • Figure 5: Himmelblau test, parameters' evolution. The population size is $N=10^5$, and the parameters $\theta$ are initially sampled uniformly from the box $[-0.5,0.5]^2$. The density is reconstructed using a 40-bin histogram in each dimension. The second row in (b) shows the trajectories of 5 randomly selected agents. The solid orange curves correspond to the SGD training dynamics, while the red dashed lines indicate the jumps induced by the PBT update after an inner training loop.
  • ...and 2 more figures

Theorems & Definitions (13)

  • Remark 1
  • Remark 2
  • Theorem 4
  • Remark 5
  • Remark 6
  • Lemma 8
  • Theorem 10
  • Remark 11
  • Definition 12: Weak measure solution
  • Lemma 13: Concentration rate
  • ...and 3 more