When Dynamic Data Selection Meets Data Augmentation

TL;DR

The paper addresses the trade-off between training efficiency and data diversity by unifying dynamic data selection with data augmentation. It introduces a joint selection mechanism based on a density distribution $p_\rho$ and a multimodal semantic-consistency distribution $p_{con}$ (via CLIP-adapters), forming $p_{sel}=p_\rho p_{con}$ to guide per-epoch sample augmentation. The method uses online HNSW for density estimation and TrivialAugment for lightweight augmentation, achieving significant training-cost reductions (e.g., on ImageNet-1k, around 40% savings) while preserving or slightly improving accuracy, and it demonstrates robustness to noisy data and generalization across architectures (ResNet, ViT, Swin). The work provides practical impact for large-scale teaching and deployment, showing the value of jointly optimizing data selection and augmentation to exploit their synergies in real-world scenarios. All mathematical components are presented with explicit formulations and are designed to scale to large datasets with modest online overhead.

Abstract

Dynamic data selection aims to accelerate training with lossless performance. However, reducing training data inherently limits data diversity, potentially hindering generalization. While data augmentation is widely used to enhance diversity, it is typically not optimized in conjunction with selection. As a result, directly combining these techniques fails to fully exploit their synergies. To tackle the challenge, we propose a novel online data training framework that, for the first time, unifies dynamic data selection and augmentation, achieving both training efficiency and enhanced performance. Our method estimates each sample's joint distribution of local density and multimodal semantic consistency, allowing for the targeted selection of augmentation-suitable samples while suppressing the inclusion of noisy or ambiguous data. This enables a more significant reduction in dataset size without sacrificing model generalization. Experimental results demonstrate that our method outperforms existing state-of-the-art approaches on various benchmark datasets and architectures, e.g., reducing 50\% training costs on ImageNet-1k with lossless performance. Furthermore, our approach enhances noise resistance and improves model robustness, reinforcing its practical utility in real-world scenarios.

Figures (5)

• Figure 1: Illustration of the distribution of our selected data points using T-SNE algorithm (left) and the density histograms without (mid) and with (right) augmenting the selected data on CIFAR-10. The selection ratio is 10%.
• Figure 2: The framework of our proposed data training method: The core idea of our framework is to construct a joint distribution that integrates both the density and semantic consistency distributions, enabling the prioritization of low-density, semantically consistent samples. After augmentation, augmented sparse samples in intra-cluster regions help to fill the underrepresented spaces, while samples located around the decision boundaries between clusters differentiate the classification decision more clearly, thus improving generalization.
• Figure 3: Experimental results on Tiny-ImageNet with noisy and corrupted data using ResNet-50. The noisy ratio is 20%.
• Figure 4: The overall cost savings achieved by our method on ViT-based architectures with lossless performance. The experiment is conducted on ImageNet-1k with a 4-A100-GPU server.
• Figure 5: Visualization of the selection results on noisy Tiny-ImageNet with a 20% noise ratio. The selection ratio is 20%.