On-the-Fly Data Augmentation via Gradient-Guided and Sample-Aware Influence Estimation

TL;DR

This paper tackles the mismatch between static data augmentation and evolving model training needs by introducing SADA, a Sample-Aware Dynamic Augmentation framework. It estimates each sample's influence on optimization by projecting its gradient onto the accumulated update direction and tracks the temporal stability of this influence within a local window to adapt augmentation strength per instance. The method is lightweight, architecture-agnostic, and plug-and-play, delivering consistent gains on CIFAR, Tiny-ImageNet, ImageNet-1k, and notably on fine-grained and long-tailed datasets, with minimal training overhead. The authors also connect SADA to generalization bounds via empirical Rademacher complexity and demonstrate broad practical impact through extensive experiments across diverse datasets and backbones.

Abstract

Data augmentation has been widely employed to improve the generalization of deep neural networks. Most existing methods apply fixed or random transformations. However, we find that sample difficulty evolves along with the model's generalization capabilities in dynamic training environments. As a result, applying uniform or stochastic augmentations, without accounting for such dynamics, can lead to a mismatch between augmented data and the model's evolving training needs, ultimately degrading training effectiveness. To address this, we introduce SADA, a Sample-Aware Dynamic Augmentation that performs on-the-fly adjustment of augmentation strengths based on each sample's evolving influence on model optimization. Specifically, we estimate each sample's influence by projecting its gradient onto the accumulated model update direction and computing the temporal variance within a local training window. Samples with low variance, indicating stable and consistent influence, are augmented more strongly to emphasize diversity, while unstable samples receive milder transformations to preserve semantic fidelity and stabilize learning. Our method is lightweight, which does not require auxiliary models or policy tuning. It can be seamlessly integrated into existing training pipelines as a plug-and-play module. Experiments across various benchmark datasets and model architectures show consistent improvements of SADA, including +7.3\% on fine-grained tasks and +4.3\% on long-tailed datasets, highlighting the method's effectiveness and practicality.

Figures (5)

• Figure 1: Evolution of Sample Difficulty Across Training Epochs. The distribution of sample difficulty evolves dynamically throughout training. Over time, a growing proportion of samples becomes easier (higher inverse sample difficulty values), particularly in later epochs, indicating a continuous shift in difficulty distribution during training. This dynamic trend highlights the necessity of dynamic and sample-aware augmentation strategies during training.
• Figure 2: Overview of Gradient-Guided On-the-Fly Data Augmentation. At epoch $t$, we quantify the sample's influence on model optimization updates and estimate its stability. The augmentation strength is then adaptively adjusted based on this interplay between model training progress and sample difficulty.
• Figure 3: The stability of our method on the two parameters, i.e., the window size and the decay factor, with CIFAR-100 using ResNet-18.
• Figure 4: Comparison in the effectiveness-efficiency tradeoff. We report the average per-epoch training costs using a 2-NVIDIA-RTX2080TI-GPUs server.
• Figure 6: Test accuracy (%) on fine-grained datasets with ResNet-50.