DNAD: Differentiable Neural Architecture Distillation

TL;DR

DNAD targets efficient neural architecture search by coupling a differentiable, topology-unconstrained search (SNPS) with knowledge-distillation regularization (DNAD). SNPS progressively prunes a dense super-network to derive a Pareto set of architectures with varying compute, while DNAD integrates activation-based KD to stabilize one-level DARTS and improve generalization in unconstrained search spaces. Empirical results on CIFAR-10 and ImageNet show competitive Pareto fronts, with DNAD achieving top-1 error of 23.7% on ImageNet at 598M FLOPs, outperforming many DARTS-based methods, and robust transfer to SVHN. The framework demonstrates that intermediate feature-based supervision from a teacher mitigates overfitting, encourages diverse, learnable operators, and yields architectures suitable for real-world resource constraints.

Abstract

To meet the demand for designing efficient neural networks with appropriate trade-offs between model performance (e.g., classification accuracy) and computational complexity, the differentiable neural architecture distillation (DNAD) algorithm is developed based on two cores, namely search by deleting and search by imitating. Primarily, to derive neural architectures in a space where cells of the same type no longer share the same topology, the super-network progressive shrinking (SNPS) algorithm is developed based on the framework of differentiable architecture search (DARTS), i.e., search by deleting. Unlike conventional DARTS-based approaches which yield neural architectures with simple structures and derive only one architecture during the search procedure, SNPS is able to derive a Pareto-optimal set of architectures with flexible structures by forcing the dynamic super-network shrink from a dense structure to a sparse one progressively. Furthermore, since knowledge distillation (KD) has shown great effectiveness to train a compact network with the assistance of an over-parameterized model, we integrate SNPS with KD to formulate the DNAD algorithm, i.e., search by imitating. By minimizing behavioral differences between the super-network and teacher network, the over-fitting of one-level DARTS is avoided and well-performed neural architectures are derived. Experiments on CIFAR-10 and ImageNet classification tasks demonstrate that both SNPS and DNAD are able to derive a set of architectures which achieve similar or lower error rates with fewer parameters and FLOPs. Particularly, DNAD achieves the top-1 error rate of 23.7% on ImageNet classification with a model of 6.0M parameters and 598M FLOPs, which outperforms most DARTS-based methods.

Figures (10)

• Figure 1: A visual interpretation for DNAD. Both the teacher network and the student super-network are divided into $3$ blocks. In the forward process, both the teacher network and the student super-network receive the same input, and the intermediate feature maps of the teacher are used as another supervision signals, which aim to guide the optimization of the dynamic super-network and to regularize the one-level optimization of DARTS.
• Figure 2: Two neural architectures searched by DNAD on CIFAR-10. In this figure, neural architectures are named in an increasing sequence of model size. Nodes with different colors denote the feature maps of neural networks. In particular, green nodes denote the inputs and the outputs. Blue nodes denote the intermediate nodes of normal cells. Red nodes denote the intermediate nodes of reduction cells. Cream yellow nodes denote the output nodes of both normal cells and reduction cells. There are no extra nodes with different colors to denote the input nodes of cells, since the inputs of each cell are actually the outputs of its former two cells. Solid lines with different colors denote different types of operators. Skip-connect is marked in red line, sep-conv-3 is marked in blue line and dil-conv-5 is marked in green line. The gray dotted lines denote the concatenation of intermediate nodes in the channel dimension.
• Figure 3: The intermediate search procedure of DNAD. The top sub-figure presents the evolutions of multiplicative sparsity coefficient $\gamma$ and the additive sparsity coefficient $\mu$. The bottom sub-figure presents the evolution of sparsity entropy of the super-network during the search procedure.
• Figure 4: The evolution of attention weights of different operators across the search procedure (for brevity, only cells 4, 8 and 12 are presented)
• Figure 5: Grad-CAM visualization with different KD settings