Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Growth strategies for arbitrary DAG neural architectures

Stella Douka, Manon Verbockhaven, Théo Rudkiewicz, Stéphane Rivaud, François P. Landes, Sylvain Chevallier, Guillaume Charpiat

TL;DR

The paper addresses the high cost of training large neural networks by enabling Neural Architecture Growth on arbitrary DAGs during training, guided by the notion of an expressivity bottleneck. By projecting the desired update onto the current architecture's tangent space and minimizing the residual, the method grows networks via new layers, edges, or widened nodes, with inter-train optimization to select promising expansions. It compares full greedy, bottleneck-restricted, and BIC-guided strategies and demonstrates a MNIST proof-of-concept where DAG growth achieves competitive accuracy with far fewer parameters and lower energy than grid-search NAS. The work suggests practical routes to reduce training and inference costs and highlights future extensions to growable convolutional modules and scalability considerations.

Abstract

Deep learning has shown impressive results obtained at the cost of training huge neural networks. However, the larger the architecture, the higher the computational, financial, and environmental costs during training and inference. We aim at reducing both training and inference durations. We focus on Neural Architecture Growth, which can increase the size of a small model when needed, directly during training using information from the backpropagation. We expand existing work and freely grow neural networks in the form of any Directed Acyclic Graph by reducing expressivity bottlenecks in the architecture. We explore strategies to reduce excessive computations and steer network growth toward more parameter-efficient architectures.

Growth strategies for arbitrary DAG neural architectures

TL;DR

The paper addresses the high cost of training large neural networks by enabling Neural Architecture Growth on arbitrary DAGs during training, guided by the notion of an expressivity bottleneck. By projecting the desired update onto the current architecture's tangent space and minimizing the residual, the method grows networks via new layers, edges, or widened nodes, with inter-train optimization to select promising expansions. It compares full greedy, bottleneck-restricted, and BIC-guided strategies and demonstrates a MNIST proof-of-concept where DAG growth achieves competitive accuracy with far fewer parameters and lower energy than grid-search NAS. The work suggests practical routes to reduce training and inference costs and highlights future extensions to growable convolutional modules and scalability considerations.

Abstract

Deep learning has shown impressive results obtained at the cost of training huge neural networks. However, the larger the architecture, the higher the computational, financial, and environmental costs during training and inference. We aim at reducing both training and inference durations. We focus on Neural Architecture Growth, which can increase the size of a small model when needed, directly during training using information from the backpropagation. We expand existing work and freely grow neural networks in the form of any Directed Acyclic Graph by reducing expressivity bottlenecks in the architecture. We explore strategies to reduce excessive computations and steer network growth toward more parameter-efficient architectures.
Paper Structure (10 sections, 5 figures, 1 table)

This paper contains 10 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Expressivity bottleneck.
  4. Growing an arbitrary DAG.
  5. Strategies for Growth.
  6. Experiments and Results
  7. Proof of concept.
  8. Growth Strategies.
  9. Conclusion and Future work
  10. Acknowledgments

Figures (5)

  • Figure 1: Expressivity Bottleneck
  • Figure 2: Example of DAG Network. We assume a new node added in color red with new weights $\alpha$ and $\omega$. We define pre-activities as A and post-activities as B or $x$.
  • Figure 3: Neural Architecture Growth results on MNIST using arbitrary DAG networks. We grow the architecture every 500 epochs.
  • Figure 4: Baselines on MNIST for test accuracy and number of parameters.
  • Figure 5: (a) Teacher-Student loss performance. The highlighted area represents the standard deviation over 6 runs. (b) Teacher-Student cumulative GPU energy consumption in Wh. Oracle: cost to train an architecture identical to the Teacher for the same number of epochs. The highlighted area represents the 85% IQR over 6 runs.