Optimizing Data Curation through Spectral Analysis and Joint Batch Selection (SALN)
Mohammadreza Sharifi
TL;DR
SALN introduces a spectral, per-batch data-curation technique that identifies informative samples within each batch using a Laplacian-based score derived from batch-specific features. By extracting features with a pretrained network, computing a cosine similarity matrix, forming a Laplacian $L = D - S$, and using the Fiedler vector to select samples, SALN achieves faster training and often higher accuracy than standard training, and generally outpaces Google's JEST on training time. Empirical results on Oxford-IIIT Pet and CIFAR-10 show SALN reducing training time up to ~8x and delivering gains in accuracy up to around 5 percentage points in some settings, while maintaining competitive performance on diverse datasets. Overall, SALN demonstrates a practical, batch-centric approach to data curation that enhances training efficiency and scalability for deep neural networks.
Abstract
In modern deep learning models, long training times and large datasets present significant challenges to both efficiency and scalability. Effective data curation and sample selection are crucial for optimizing the training process of deep neural networks. This paper introduces SALN, a method designed to prioritize and select samples within each batch rather than from the entire dataset. By utilizing jointly selected batches, SALN enhances training efficiency compared to independent batch selection. The proposed method applies a spectral analysis-based heuristic to identify the most informative data points within each batch, improving both training speed and accuracy. The SALN algorithm significantly reduces training time and enhances accuracy when compared to traditional batch prioritization or standard training procedures. It demonstrates up to an 8x reduction in training time and up to a 5\% increase in accuracy over standard training methods. Moreover, SALN achieves better performance and shorter training times compared to Google's JEST method developed by DeepMind.