Coresets for Data-efficient Training of Machine Learning Models

TL;DR

CRAIG introduces a general coreset-based preprocessing to accelerate Incremental Gradient methods by selecting a weighted subset that closely approximates the full gradient through a submodular facility-location objective. Theoretical results show that IG on the CRAIG subset converges at the same rate as IG on the full data, up to an additive error that scales with the gradient-approximation bound ε, yielding speedups proportional to |V|/|S|. Empirically, CRAIG delivers up to 6x speedups on convex problems and 3x on non-convex networks while maintaining comparable loss and accuracy, across logistic regression and deep nets. The method is compatible with SGD, SAGA, SVRG, and can be updated during training, making data-efficient training practical for large-scale models.

Abstract

Incremental gradient (IG) methods, such as stochastic gradient descent and its variants are commonly used for large scale optimization in machine learning. Despite the sustained effort to make IG methods more data-efficient, it remains an open question how to select a training data subset that can theoretically and practically perform on par with the full dataset. Here we develop CRAIG, a method to select a weighted subset (or coreset) of training data that closely estimates the full gradient by maximizing a submodular function. We prove that applying IG to this subset is guaranteed to converge to the (near)optimal solution with the same convergence rate as that of IG for convex optimization. As a result, CRAIG achieves a speedup that is inversely proportional to the size of the subset. To our knowledge, this is the first rigorous method for data-efficient training of general machine learning models. Our extensive set of experiments show that CRAIG, while achieving practically the same solution, speeds up various IG methods by up to 6x for logistic regression and 3x for training deep neural networks.

Figures (6)

• Figure 1: Loss residual and error rate of SGD, SVRG, SAGA for Logistic Regression on Covtype data set with 581,012 data points. We compare CRAIG (10% selected subset) (blue) vs. 10% random subset (green) vs. entire data set (orange). CRAIG gives the average speedup of 3x for achieving similar loss residual and error rate across the three optimization methods.
• Figure 2: Normed difference between the full gradient, the gradient of the subset found by CRAIG (Eq. \ref{['eq:error']}), and the theoretical upper-bound $\epsilon$ (Eq. \ref{['eq:fl_min']}). The values are normalized by the largest full gradient norm. The transparent green lines demonstrate various random subsets, and the opaque green line shows their average.
• Figure 3: Training loss residual for SGD applied to subsets of size $10\%, 20\%, \cdots, 90\%$ found by CRAIG vs. random subsets of the same size from Ijcnn1. We get 5.6x speedup from applying SGD to subset of size 30% compared to the entire dataset.
• Figure 4: Test accuracy and training loss of SGD applied to subsets found by CRAIG vs. random subsets on MNIST with a 2-layer neural network. CRAIG provides 2x to 3x speedup and a better generalization performance.
• Figure 5: Test accuracy vs. fraction of data selected during training of ResNet-20 on CIFAR10. (a) At the beginning of ever epoch, a new subset of size 1%, 2%, 3%, 4%, 5%, 10%, or 20% is selected by CRAIG. (b) Every 5 epochs a new subset of similar size is selected by CRAIG. SGD is then applied to training on the selected subsets. The x-axis shows the fraction of training data points that are used by SGD during the training process. Note that for a given subset size, backpropagation is done on the same number of data points for CRAIG and random. However, CRAIG selects a smaller number of distinct data points during the training. Therefore, CRAIG is data-efficient for training neural networks.

Theorems & Definitions (4)

• Theorem 1
• Theorem 2
• Lemma 3
• proof