A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances

TL;DR

This survey unifies the diverse literature on coreset selection by organizing methods into training-free, training-oriented, and blind approaches, and by introducing submodular, bilevel, and pseudo-labeling perspectives. It analyzes how pruning and dataset structure influence generalization and neural scaling, and compares methods across computation, robustness, and performance under varying constraints. By detailing concrete algorithms (e.g., Herding, k-Center Greedy, GraNd, DeepFool, FASS, PRISM, SIMILAR, CRAIG, GradMatch, RETRIEVE, GLISTER, ELFS, ZCore) and their trade-offs, the work highlights when to apply each approach and exposes open challenges like outlier handling and foundation-model adaptation. The paper also surveys a wide range of applications—from image enhancement and NAS to dataset distillation, continual learning, RL, unlearning, SSL, and LLM pretraining—demonstrating the practical impact of principled data reduction on modern AI systems. Overall, coreset selection emerges as a nontrivial yet powerful tool to improve efficiency and generalization in large-scale learning, with future directions pointing toward adaptive, fair, and multi-modal data pruning frameworks.

Abstract

Coreset selection targets the challenge of finding a small, representative subset of a large dataset that preserves essential patterns for effective machine learning. Although several surveys have examined data reduction strategies before, most focus narrowly on either classical geometry-based methods or active learning techniques. In contrast, this survey presents a more comprehensive view by unifying three major lines of coreset research, namely, training-free, training-oriented, and label-free approaches, into a single taxonomy. We present subfields often overlooked by existing work, including submodular formulations, bilevel optimization, and recent progress in pseudo-labeling for unlabeled datasets. Additionally, we examine how pruning strategies influence generalization and neural scaling laws, offering new insights that are absent from prior reviews. Finally, we compare these methods under varying computational, robustness, and performance demands and highlight open challenges, such as robustness, outlier filtering, and adapting coreset selection to foundation models, for future research.

Figures (12)

• Figure 1: Venn diagram of coreset selection and active learning. Coreset selection typically assumes a labeled dataset, whereas active learning uses small labeled datasets and a large unlabeled pool. Many techniques are shared: both coreset selection and active learning rely on criteria (e.g., uncertainty, diversity, and coverage) to select data subsets.
• Figure 2: Taxonomy used in this survey. We differentiate between training-free and training-oriented methods.
• Figure 3: The active learning pipeline. The idea of active learning is to prune a large pool of unlabeled dataset $\mathcal{U}$ and to extend the already existing labeled dataset with essential, newly annotated data.
• Figure 4: Illustration of the iterative selection of Herding. As the angle between the direction to the mean of the real dataset $\mathcal{T}$ and to $\mathbf{x}_1$ is smaller than to $\mathbf{x}_2$, as well as the distance to $\mathbf{m}_{t-1}$ is higher to $\mathbf{x}_1$, $\mathbf{x}_1$ will be included for the iteration step $t$, leading to a new coreset mean $\mathbf{m}_t$.
• Figure 5: Coreset selection performance of Herding and k-Center Greedy on CIFAR-10 for different feature embeddings (ResNet-18 he2016deep, InceptionNet szegedy2015going, and VGG simonyan2014very).