Chebyshev Moment Regularization (CMR): Condition-Number Control with Moment Shaping

TL;DR

CMR addresses the problem of brittle optimization in deep networks caused by poorly conditioned layer spectra by directly regularizing spectral geometry. It combines a log-condition proxy for spectral edges with Chebyshev moments to shape the interior, using a decoupled, capped mixing that preserves task gradients. Theoretical results show monotone descent of the condition proxy, bounded moment gradients, and orthogonal invariance, while experiments on κ-stress MNIST demonstrate dramatic conditioning improvements and test accuracy recovery. The approach acts as an optimization-driven spectral preconditioner with practical applicability across architectures and optimizers, offering a direct route to stable, accurate learning.

Abstract

We introduce \textbf{Chebyshev Moment Regularization (CMR)}, a simple, architecture-agnostic loss that directly optimizes layer spectra. CMR jointly controls spectral edges via a log-condition proxy and shapes the interior via Chebyshev moments, with a decoupled, capped mixing rule that preserves task gradients. We prove strictly monotone descent for the condition proxy, bounded moment gradients, and orthogonal invariance. In an adversarial ``$κ$-stress'' setting (MNIST, 15-layer MLP), \emph{compared to vanilla training}, CMR reduces mean layer condition numbers by $\sim\!10^3$ (from $\approx3.9\!\times\!10^3$ to $\approx3.4$ in 5 epochs), increases average gradient magnitude, and restores test accuracy ( $\approx10\%\!\to\!\approx86\%$ ). These results support \textbf{optimization-driven spectral preconditioning}: directly steering models toward well-conditioned regimes for stable, accurate learning.

Figures (8)

• Figure 2: Final Test Accuracy Across Baselines. This bar chart illustrates the final test accuracy achieved by different regularization methods after 20 training epochs. CMR achieves competitive accuracy, slightly outperforming vanilla and matching or exceeding L2 and SN. This indicates that the spectral benefits of CMR are gained without compromising the model's primary task performance.
• Figure 3: Evolution of Median Layer Condition Number ($\kappa(G)$) Over Training. The median condition number of the Gram matrices for each layer is plotted over 20 epochs on a logarithmic scale. The vanilla model's conditioning rapidly deteriorates, increasing significantly after epoch 4. In contrast, CMR effectively stabilizes and reduces the median $\kappa(G)$, maintaining a consistently lower (i.e., better) value throughout training. This clearly illustrates CMR's ability to maintain well-conditioned layer operations.
• Figure 4: Evolution of 90th-Percentile Layer Condition Number ($\kappa(G)$) Over Training. This plot, on a logarithmic scale, shows the 90th-percentile condition number, highlighting the behavior of the most ill-conditioned layers. Similar to the median, the vanilla baseline experiences a substantial increase in its P90 $\kappa(G)$, indicating that a significant portion of its layers become severely ill-conditioned. CMR, however, effectively mitigates this rise, keeping the P90 $\kappa(G)$ remarkably stable and low, demonstrating robust control over even the "worst-case" layers in the network.
• Figure : (a) Test accuracy
• Figure : (a) Test accuracy

Theorems & Definitions (15)

• theorem 1: Monotone Descent for the Condition Proxy
• corollary 1: Control over the Log-Condition Number
• lemma 1: Moment Gradients are Bounded and Scale-Friendly
• remark 1
• proposition 1: Orthogonal Invariance
• proposition 2: Layerwise Jacobian bound.
• proof
• corollary 2: Gradient propagation
• proof
• lemma 2: Gradient of $\rho_{\text{cond}}$