Automatic Stability and Recovery for Neural Network Training

TL;DR

This work tackles training instability in modern neural networks by introducing a runtime stability controller that sits above the optimizer and uses an external innovation signal from secondary measurements to decide whether to accept updates or rollback. By treating training as a controlled stochastic process and decoupling stability monitoring from the optimizer, it provides runtime safety guarantees including bounded degradation and one-step recovery from destabilizing updates. The method is optimizer-agnostic, incurs modest overhead, and demonstrates robust recovery and improved reliability across vision and transformer models, including under controlled catastrophic perturbations. These findings suggest a practical, self-healing paradigm for training pipelines, reducing wasted compute and manual interventions without modifying standard optimization rules.

Abstract

Training modern neural networks is increasingly fragile, with rare but severe destabilizing updates often causing irreversible divergence or silent performance degradation. Existing optimization methods primarily rely on preventive mechanisms embedded within the optimizer, offering limited ability to detect and recover from instability once it occurs. We introduce a supervisory runtime stability framework that treats optimization as a controlled stochastic process. By isolating an innovation signal derived from secondary measurements, such as validation probes, the framework enables automatic detection and recovery from destabilizing updates without modifying the underlying optimizer. We provide theoretical runtime safety guarantees that formalize bounded degradation and recovery. Our implementation incurs minimal overhead and is compatible with memory-constrained training settings.

Figures (5)

• Figure 1: Runtime stability controller as an external supervisory layer. The optimizer proposes parameter updates based on training batches. A secondary measurement signal evaluates the proposed update, and a stability controller decides whether to accept it or trigger rollback by restoring a previously accepted safe state. The controller operates solely on the measurement signal and stored snapshots.
• Figure 2: Probe loss recovery for ResNet-18 on CIFAR-10 under a multi-step catastrophic perturbation. Curves show mean $\pm$ standard deviation over $N=20$ seeds. The controller substantially reduces both peak degradation and recovery variance.
• Figure 3: Innovation signal $\nu_t$ showing a sharp and localized deviation during the injected destabilization window. Outside this window, the signal remains tightly bounded, indicating stable behavior under nominal training dynamics. The separation between these regimes enables reliable detection of destabilizing updates without sensitivity to minibatch noise.
• Figure 4: Evolution of parameter $\ell_2$ norms during training. The controller prevents persistent drift into high-norm regimes following destabilization, indicating improved internal stability.
• Figure 5: Probe loss recovery for a character-level Transformer model under catastrophic perturbation. Results are averaged over $N=20$ seeds. The controller consistently improves recovery speed and stability.

Theorems & Definitions (7)

• Definition 5.1: Admissible Innovation Signal
• Theorem 7.3: Bounded Deviation from the Reference Signal
• proof
• Proposition 7.4: One-Step Recovery
• proof
• Proposition 7.5: Monotone Safety Envelope
• proof