Shrinkage Initialization for Smooth Learning of Neural Networks
Miao Cheng, Feiyan Zhou, Hongwei Zou, Limin Wang
TL;DR
This paper addresses the challenge of initializing neural networks to enable smooth learning across architectures with random or varied layer configurations. It introduces shrinkage initialization (SINL), a SVD-based, orthogonal-rotation method that builds cross-layer bridges $E_{ij} = X_j X_i^T (X_i X_i^T)^{+}$, computes $E_{ij} = U_{ij} S_{ij} V_{ij}^T$, and updates boundary weights with $W_{i \to i+1} \leftarrow W_{i \to i+1} V_{ij}^T$ and $W_{j-1 \to j} \leftarrow U_{ij} W_{j-1 \to j}$; median layers are handled via a reconstructed $W_{i \to j} = U_{ij} V_{ij}^T$ when needed, with complexity $O(p^2 q + p q^2)$ per bridge and $O(p q n + q^3)$ for data bridges. SINL is shown to be a generalized, low-complexity initialization that supports smooth learning and demonstrates robust performance across artificial datasets, outperforming or matching established methods (BN, LSUV, DIN) in various settings. The work highlights the practical impact of structured, rotation-based initialization for networks with random or irregular layer configurations, potentially reducing training instability and improving convergence.
Abstract
The successes of intelligent systems have quite relied on the artificial learning of information, which lead to the broad applications of neural learning solutions. As a common sense, the training of neural networks can be largely improved by specifically defined initialization, neuron layers as well as the activation functions. Though there are sequential layer based initialization available, the generalized solution to initial stages is still desired. In this work, an improved approach to initialization of neural learning is presented, which adopts the shrinkage approach to initialize the transformation of each layer of networks. It can be universally adapted for the structures of any networks with random layers, while stable performance can be attained. Furthermore, the smooth learning of networks is adopted in this work, due to the diverse influence on neural learning. Experimental results on several artificial data sets demonstrate that, the proposed method is able to present robust results with the shrinkage initialization, and competent for smooth learning of neural networks.