Self-Expanding Neural Networks

TL;DR

Self-Expanding Neural Networks (SENN) tackle the challenge of selecting neural network capacity during training by growing width and depth only when beneficial, without restarting optimization. They introduce the natural expansion score $\eta = \mathbf{g}^\top \mathbf{F}^{-1} \mathbf{g}$ to quantify potential loss reduction from additions, and use a principled, Kronecker-factored Fisher approximation to efficiently bound and guide growth. The framework provides four ingredients: how to add capacity without changing the function, initialization to maximize $\Delta \eta$, where to add capacity, and when to add capacity, forming a cohesive growth strategy that adapts to data complexity. Extensive experiments across regression, 2-D classification, and image tasks demonstrate stable, continuous expansion and occasional pruning, with benefits in transfer learning from pre-trained models. Overall, SENN offers a scalable, principled approach to dynamic architectures that tailor themselves to task difficulty and data-driven capacity needs, with potential extensions to transformers and continual-learning scenarios.

Abstract

The results of training a neural network are heavily dependent on the architecture chosen; and even a modification of only its size, however small, typically involves restarting the training process. In contrast to this, we begin training with a small architecture, only increase its capacity as necessary for the problem, and avoid interfering with previous optimization while doing so. We thereby introduce a natural gradient based approach which intuitively expands both the width and depth of a neural network when this is likely to substantially reduce the hypothetical converged training loss. We prove an upper bound on the ``rate'' at which neurons are added, and a computationally cheap lower bound on the expansion score. We illustrate the benefits of such Self-Expanding Neural Networks with full connectivity and convolutions in both classification and regression problems, including those where the appropriate architecture size is substantially uncertain a priori.

Figures (6)

• Figure 1: SENN can add width (orange) and depth (green) to a neural network without changing the overall function.
• Figure 2: A single layer SENN (black, solid) is trained on a target function (red, dashed) via least-squares regression on samples (blue, markers). Vertical lines show the location of existing neurons. The lower panels show $\Delta \eta' / \eta_0$ as a function of the location and scale of the nonlinearity introduced by a new neuron. Accepted and rejected proposals are marked in red and black respectively. From left to right we see the landscape before and immediately after the fourth neuron is added, before the fifth is added, and at the end of training. SENN adds neurons where relevant in order to achieve a good fit.
• Figure 3: 2-D binary classification with SENN. The normalized layer addition score $\Delta \eta' / \eta_c$ is shown as a function of optimization steps; the horizontal bar shows the point above which a layer will be added. The score increases during three phases during which the SENN has initial zero, one and then two hidden layers. These layer insertions allow SENN to represent more complex decision boundaries (shown at the top) when required for global expressivity.
• Figure 4: SENN shows reasonable and reproducible hidden layer growth on MNIST at appealing any-time validation accuracy without intermittent perturbations (left pair of panels). SENN features appropriate scaling with respect to data complexity in its chosen network sizes (right panel).
• Figure 5: SENN stabilizes easily, even when perturbed by a cyclic cosine learning rate schedule. As can be observed in the last two layers of block 1, the network capacity expands and compresses in correspondence with the learning rate.

Theorems & Definitions (17)

• Theorem 3.1: Upper bound on the "rate" of neuron addition
• Theorem 3.2: Computationally cheap lower bound on increase in natural expansion score within a layer l
• Lemma 1.1
• proof
• Lemma 1.2
• proof
• Theorem 1.3
• proof
• Lemma 1.4
• proof