Growing Efficient Accurate and Robust Neural Networks on the Edge

TL;DR

This work tackles the challenge of robust and efficient Deep Neural Network training on resource-constrained Edge devices, aiming to obviate cloud-based training and data transmission. It introduces GEARnn, a two-phase approach that grows compact networks via One-Shot Growth (OSG) and robustifies them with Efficient Robust Augmentation (ERA); a 2-Phase variant (GEARnn-2) initializes from clean-data growth and then trains with augmented data to achieve robust accuracy efficiently. Across CIFAR-10/100 and Tiny ImageNet on architectures like VGG-19 and MobileNet, GEARnn-2 yields higher robust accuracy at equal final size and markedly reduces training time and energy, with Jetson edge results showing comparable clean accuracy to private baselines while cutting training cost by about 2–3x. The approach highlights that edge-only growth with robust augmentation is viable, enabling privacy-preserving, energy-efficient edge learning suitable for real-world deployments.

Abstract

The ubiquitous deployment of deep learning systems on resource-constrained Edge devices is hindered by their high computational complexity coupled with their fragility to out-of-distribution (OOD) data, especially to naturally occurring common corruptions. Current solutions rely on the Cloud to train and compress models before deploying to the Edge. This incurs high energy and latency costs in transmitting locally acquired field data to the Cloud while also raising privacy concerns. We propose GEARnn (Growing Efficient, Accurate, and Robust neural networks) to grow and train robust networks in-situ, i.e., completely on the Edge device. Starting with a low-complexity initial backbone network, GEARnn employs One-Shot Growth (OSG) to grow a network satisfying the memory constraints of the Edge device using clean data, and robustifies the network using Efficient Robust Augmentation (ERA) to obtain the final network. We demonstrate results on a NVIDIA Jetson Xavier NX, and analyze the trade-offs between accuracy, robustness, model size, energy consumption, and training time. Our results demonstrate the construction of efficient, accurate, and robust networks entirely on an Edge device.

Figures (10)

• Figure 1: Improvements in robust accuracy, training time, and model size (area of circles) of our proposed GEARnn method measured on NVIDIA Jetson Xavier NX Edge device jetson. Robust accuracy is evaluated on CIFAR-10-C for GEARnn, full network baselines (VGG-19), and small network baselines (5% VGG-19 networks with same topology as GEARnn-2). For robust training, we employ AugMix hendrycks2019augmix. GEARnn demonstrates significant reduction in training complexity over robust baselines at similar robust and clean accuracies (shown in \ref{['subsec: Jetson results']}).
• Figure 2: Proposed approach: GEARnn-1 performs One-Shot Growth (OSG) on augmented data ($\mathcal{D}_{\text{aug}}$) generated by Efficient Robust Augmentation (ERA) (using clean data ($\mathcal{D}_{\text{in}}$)) in a single phase (1-Phase). GEARnn-2 performs OSG using $\mathcal{D}_{\text{in}}$ first followed by parametric training on $\mathcal{D}_{\text{aug}}$ in two consecutive phases (2-phase). Here $\mathcal{L}_{\text{CE}}$ and $\mathcal{L}_{\text{aug}}$ denote the cross-entropy loss and augmented loss, respectively.
• Figure 3: OSG takes in labeled data ($\mathcal{D}$) and backbone network $f_0$, and performs a training step, a growth step, and a training step in sequence to generate network $f_2$. The 2-tuple $(\mathcal{L}, \mathcal{E})=$ (loss function, number of epochs) employed in each step.
• Figure 4: ERA takes in clean data ($\mathcal{D}_{\text{in}}$) as input and applies a set of stochastic transforms to generate augmented data ($\mathcal{D}_{\text{aug}}$) in an efficient manner.
• Figure 5: GEARnn-2 achieves higher robustness at the same: (a) number of robust training epochs at final model size, and (b) training time, for VGG-19/CIFAR-100 on Quadro.