When To Grow? A Fitting Risk-Aware Policy for Layer Growing in Deep Neural Networks

TL;DR

This work tackles the problem of when to grow neural networks during training, revealing that neural growth induces a regularization effect whose strength depends on growth timing. It introduces FRAGrow, a fitting-risk-aware growth policy that uses the Overfitting Risk Level (ORL) to adapt growth speed via $I = \frac{I_{max}}{1 + e^{\alpha - ORL}}$ with $I_{max} = \frac{E_T - E^{min}_{F}}{n}$, balancing underfitting and overfitting risks. Through CIFAR-10/100 and ImageNet experiments across VGG, ResNet, and MobileNetV2, FRAGrow yields up to about 1.3 percentage points improvement for underfitting cases and maintains competitive accuracy for overfitting cases while reducing training time, compared with traditional periodic, convergent, or Lipgrow policies. The findings highlight the importance of growth-timing decisions in regularization dynamics and pave the way for broader applications in vision tasks with more efficient growth-based training.

Abstract

Neural growth is the process of growing a small neural network to a large network and has been utilized to accelerate the training of deep neural networks. One crucial aspect of neural growth is determining the optimal growth timing. However, few studies investigate this systematically. Our study reveals that neural growth inherently exhibits a regularization effect, whose intensity is influenced by the chosen policy for growth timing. While this regularization effect may mitigate the overfitting risk of the model, it may lead to a notable accuracy drop when the model underfits. Yet, current approaches have not addressed this issue due to their lack of consideration of the regularization effect from neural growth. Motivated by these findings, we propose an under/over fitting risk-aware growth timing policy, which automatically adjusts the growth timing informed by the level of potential under/overfitting risks to address both risks. Comprehensive experiments conducted using CIFAR-10/100 and ImageNet datasets show that the proposed policy achieves accuracy improvements of up to 1.3% in models prone to underfitting while achieving similar accuracies in models suffering from overfitting compared to the existing methods.

Figures (3)

• Figure 1: When to grow policy. To expedite the training process of the target large model with size $S_e$, neural growth grows the initial small model from size $S_0$ at time $t_0$ into $S_e$ at time $t_e$. For the prevailing conventional strategies (illustrated by the green curve), $t_e$ is not influenced by overfitting or underfitting risks directly, and these strategies may fail to address under/over fitting risks effectively. By contrast, our fitting risk-aware policy accelerates growth (black curve) by reducing $t_e$ to mitigate underfitting risks when the target model exhibits underfitting tendencies. Conversely, it decelerates growth (red curve) by extending the $t_e$ for models displaying signs of overfitting, addressing the overfitting risk.
• Figure 2: Comparison of vanilla training and neural growth. In standard training (vanilla), all layers undergo training for $E_1$ epochs. However, in the context of neural growth, only layer $B_1$ is trained for $E_1$ epochs, while the majority of layers ($B_2$ and $B_3$) are subjected to reduced training epochs ($E_2$ and $E_3$ respectively).
• Figure 3: Regularization effect of neural growth on CIFAR100 dataset. In this experiment, the network is grown periodically, and new network layers are added after each growth interval. The growth speed is measured by the inverse of the growth interval. Increasing growth speed leads to larger average training epochs $\bar{E}$ and a weaker regularization effect, resulting in smaller training errors.