Improving Resnet-9 Generalization Trained on Small Datasets

TL;DR

The paper tackles the problem of achieving high image-classification accuracy from a very small dataset under tight time constraints, targeting edge-enabled training. It proposes a compact ResNet-9 model augmented with a suite of techniques—Sharpness Aware Minimization, Gradient Centralization, Improved Preprocessing (including Label Smoothing, Weight Decay, CELU activation, and Input Patch Whitening), and Meta-learning based Training—to boost generalization. Empirical results on a 5,000-image CIFAR-10 subset show substantial gains over the baseline, with the best performance achieved when combining SAM and IP, and additional improvements from GC and meta-learning. The findings suggest these methods are largely orthogonal and can be integrated to enable fast, generalizable learning on small datasets, which is practical for edge deployments and hardware-aware training scenarios.

Abstract

This paper presents our proposed approach that won the first prize at the ICLR competition on Hardware Aware Efficient Training. The challenge is to achieve the highest possible accuracy in an image classification task in less than 10 minutes. The training is done on a small dataset of 5000 images picked randomly from CIFAR-10 dataset. The evaluation is performed by the competition organizers on a secret dataset with 1000 images of the same size. Our approach includes applying a series of technique for improving the generalization of ResNet-9 including: sharpness aware optimization, label smoothing, gradient centralization, input patch whitening as well as metalearning based training. Our experiments show that the ResNet-9 can achieve the accuracy of 88% while trained only on a 10% subset of CIFAR-10 dataset in less than 10 minuets

Figures (1)

• Figure 1: Meta-learning Algorithm