Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LION-DG: Layer-Informed Initialization with Deep Gradient Protocols for Accelerated Neural Network Training

Hyunjun Kim

TL;DR

LION-DG introduces a layer-informed initialization that zero-initializes auxiliary heads in deeply-supervised networks while applying standard He-initialization to the backbone. The authors prove gradient decoupling at initialization and describe a gradient awakening dynamic where auxiliary gradients gradually emerge as auxiliary weights grow, delivering an implicit warmup without hyperparameters. Empirically, LION-DG yields consistent convergence speedups on concatenative architectures (e.g., DenseNet-DS) and competitive accuracy, with the best overall performance achieved when combined with LSUV in a Hybrid setup. The findings offer architecture-aware guidelines for faster training of multi-head networks and highlight zero-cost warmup as a practical benefit for real-world training efficiency.

Abstract

Weight initialization remains decisive for neural network optimization, yet existing methods are largely layer-agnostic. We study initialization for deeply-supervised architectures with auxiliary classifiers, where untrained auxiliary heads can destabilize early training through gradient interference. We propose LION-DG, a layer-informed initialization that zero-initializes auxiliary classifier heads while applying standard He-initialization to the backbone. We prove that this implements Gradient Awakening: auxiliary gradients are exactly zero at initialization, then phase in naturally as weights grow -- providing an implicit warmup without hyperparameters. Experiments on CIFAR-10 and CIFAR-100 with DenseNet-DS and ResNet-DS architectures demonstrate: (1) DenseNet-DS: +8.3% faster convergence on CIFAR-10 with comparable accuracy, (2) Hybrid approach: Combining LSUV with LION-DG achieves best accuracy (81.92% on CIFAR-10), (3) ResNet-DS: Positive speedup on CIFAR-100 (+11.3%) with side-tap auxiliary design. We identify architecture-specific trade-offs and provide clear guidelines for practitioners. LION-DG is simple, requires zero hyperparameters, and adds no computational overhead.

LION-DG: Layer-Informed Initialization with Deep Gradient Protocols for Accelerated Neural Network Training

TL;DR

LION-DG introduces a layer-informed initialization that zero-initializes auxiliary heads in deeply-supervised networks while applying standard He-initialization to the backbone. The authors prove gradient decoupling at initialization and describe a gradient awakening dynamic where auxiliary gradients gradually emerge as auxiliary weights grow, delivering an implicit warmup without hyperparameters. Empirically, LION-DG yields consistent convergence speedups on concatenative architectures (e.g., DenseNet-DS) and competitive accuracy, with the best overall performance achieved when combined with LSUV in a Hybrid setup. The findings offer architecture-aware guidelines for faster training of multi-head networks and highlight zero-cost warmup as a practical benefit for real-world training efficiency.

Abstract

Weight initialization remains decisive for neural network optimization, yet existing methods are largely layer-agnostic. We study initialization for deeply-supervised architectures with auxiliary classifiers, where untrained auxiliary heads can destabilize early training through gradient interference. We propose LION-DG, a layer-informed initialization that zero-initializes auxiliary classifier heads while applying standard He-initialization to the backbone. We prove that this implements Gradient Awakening: auxiliary gradients are exactly zero at initialization, then phase in naturally as weights grow -- providing an implicit warmup without hyperparameters. Experiments on CIFAR-10 and CIFAR-100 with DenseNet-DS and ResNet-DS architectures demonstrate: (1) DenseNet-DS: +8.3% faster convergence on CIFAR-10 with comparable accuracy, (2) Hybrid approach: Combining LSUV with LION-DG achieves best accuracy (81.92% on CIFAR-10), (3) ResNet-DS: Positive speedup on CIFAR-100 (+11.3%) with side-tap auxiliary design. We identify architecture-specific trade-offs and provide clear guidelines for practitioners. LION-DG is simple, requires zero hyperparameters, and adds no computational overhead.
Paper Structure (48 sections, 7 theorems, 22 equations, 2 figures, 8 tables, 1 algorithm)

This paper contains 48 sections, 7 theorems, 22 equations, 2 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Network Initialization
  4. Deeply-Supervised Architectures
  5. Multi-Task Gradient Balancing
  6. Gradient Warmup Strategies
  7. Zero-Initialization Techniques
  8. Method: LION-DG
  9. Problem Setup
  10. LION-DG Initialization
  11. Theoretical Analysis
  12. Gradient Decoupling at Initialization
  13. Gradient Awakening Dynamics
  14. Comparison with Explicit Warmup
  15. Architecture Dependence
  16. ...and 33 more sections

Key Result

Proposition 1

When $W_{\text{aux}}^{(\ell)} = 0$, the gradient of the auxiliary loss with respect to backbone parameters is exactly zero at initialization:

Figures (2)

  • Figure 1: Left: Training and validation accuracy curves on CIFAR-10 DenseNet-DS. LION-DG reaches 70% training accuracy faster than He-init baseline. Right: Gradient ratio (aux/main) over training epochs, showing the "awakening" dynamics where auxiliary gradients gradually increase their contribution.
  • Figure 2: Validation accuracy comparison across initialization methods on CIFAR-10. DenseNet-DS and ResNet-DS show different preferences: ResNet-DS favors LSUV while DenseNet-DS benefits most from the Hybrid approach.

Theorems & Definitions (13)

  • Proposition 1: Gradient Decoupling
  • proof
  • Proposition 2: Linear Weight Growth
  • proof
  • Corollary 1: Implicit vs. Explicit Warmup
  • Theorem 1: Concatenative vs. Additive Residual Paths
  • proof : Proof Sketch
  • Proposition 3: Gradient Decoupling
  • proof
  • Proposition 4: Weight Growth
  • ...and 3 more