LDLT L-Lipschitz Network Weight Parameterization Initialization

TL;DR

This paper derives the exact marginal output variance for LDLT-based L-Lipschitz layers under Gaussian initialization by modeling the weight Gramian S as a Wishart matrix and employing James' zonal-polynomial framework along with a Laplace integral representation. By expanding matrix moments via Wick/Isserlis recurrences, the authors obtain truncated series up to k = 10 that accurately approximate variance for small-to-moderate σ^2 and validate these results with Monte Carlo simulations. They further analyze variance scaling and gradient propagation, revealing that large σ^2 can restore forward variance but degrade backward-gradient dynamics, and show that unit-variance initialization is not attainable in deep settings under the LDLT scheme. Empirical evaluation on the Higgs dataset demonstrates that, although the theory provides variance-preserving guidelines, He/Glorot-like initializations remain competitive or superior in practice when combined with modern optimizers like AdamW, highlighting a gap between theory and real-world performance and informing practical initialization choices for deep Lipschitz networks.

Abstract

We analyze initialization dynamics for LDLT-based $\mathcal{L}$-Lipschitz layers by deriving the exact marginal output variance when the underlying parameter matrix $W_0\in \mathbb{R}^{m\times n}$ is initialized with IID Gaussian entries $\mathcal{N}(0,σ^2)$. The Wishart distribution, $S=W_0W_0^\top\sim\mathcal{W}_m(n,σ^2 \boldsymbol{I}_m)$, used for computing the output marginal variance is derived in closed form using expectations of zonal polynomials via James' theorem and a Laplace-integral expansion of $(α\boldsymbol{I}_m+S)^{-1}$. We develop an Isserlis/Wick-based combinatorial expansion for $\operatorname{\mathbb{E}}\left[\operatorname{tr}(S^k)\right]$ and provide explicit truncated moments up to $k=10$, which yield accurate series approximations for small-to-moderate $σ^2$. Monte Carlo experiments confirm the theoretical estimates. Furthermore, empirical analysis was performed to quantify that, using current He or Kaiming initialization with scaling $1/\sqrt{n}$, the output variance is $0.41$, whereas the new parameterization with $10/ \sqrt{n}$ for $α=1$ results in an output variance of $0.9$. The findings clarify why deep $\mathcal{L}$-Lipschitz networks suffer rapid information loss at initialization and offer practical prescriptions for choosing initialization hyperparameters to mitigate this effect. However, using the Higgs boson classification dataset, a hyperparameter sweep over optimizers, initialization scale, and depth was conducted to validate the results on real-world data, showing that although the derivation ensures variance preservation, empirical results indicate He initialization still performs better.

Figures (17)

• Figure 1: Variance difference estimation for weight parameterization sizes from 2 to 9
• Figure 2: Variance difference estimation for weight parameterization sizes from 10 to 90
• Figure 3: Variance difference estimation for weight parameterization sizes from 100 to 900
• Figure 4: Relationship between the output variance and the normalized input variance using $n$ scaling at multiple $W_0 \in \mathbb{R}^{n \times n}$ dimensions with $\alpha = \gamma = 1$
• Figure 5: Relationship between the output variance and the normalized input variance using $\sqrt{n}$ scaling at multiple $W_0 \in \mathbb{R}^{n \times n}$ dimensions with $\alpha = \gamma = 1$

Theorems & Definitions (8)

• Lemma 3.1
• Lemma 3.2
• Theorem 3.3
• Lemma 3.4
• Lemma 3.5
• proof
• Theorem 4.1
• Remark 1: Gradient Trade-off