MODE: Multi-Objective Adaptive Coreset Selection

TL;DR

MODE introduces a dynamic, multi-objective approach to coreset selection that adapts the data subset during training by weighting four complementary strategies (uncertainty, diversity, class balance, and boundary proximity). By modeling training state and learning strategy weights with a meta-controller, MODE achieves a $(1-1/e)$-approximation guarantee for fixed weights and converges the strategy weights at a rate of $O(1/\sqrt{t})$, while maintaining practical $O(K \cdot n \log n)$ time and memory efficiency. The framework supports selective recomputation to further reduce overhead, and its strategy weights reveal interpretable insight into data utility evolution across training phases. Empirical results across CIFAR, Fashion-MNIST, SVHN, and ImageNet subsets show MODE achieves competitive accuracy under tight budgets, with notable gains at low data regimes and substantial reductions in memory usage, enabling scalable, privacy-aware, and explainable data selection for large-scale learning.

Abstract

We present Mode(Multi-Objective adaptive Data Efficiency), a framework that dynamically combines coreset selection strategies based on their evolving contribution to model performance. Unlike static methods, \mode adapts selection criteria to training phases: emphasizing class balance early, diversity during representation learning, and uncertainty at convergence. We show that MODE achieves (1-1/e)-approximation with O(n \log n) complexity and demonstrates competitive accuracy while providing interpretable insights into data utility evolution. Experiments show \mode reduces memory requirements

Figures (8)

• Figure 1: Illustration of MODE components.
• Figure 2: MODE achieves 74-78% of the theoretical worst-case bound across budgets on CIFAR-10, with approximation quality following the predicted $O(1/\sqrt{B})$ scaling.
• Figure 3: Performance metrics for 25k budget
• Figure 4: Final strategy weight distribution across different budget constraints. The heatmap shows how MODE adaptively allocates importance to different strategies based on available resources
• Figure 5: Temperature evolution for varying budgets, balancing exploration and exploitation.

Theorems & Definitions (13)

• Theorem 1: Approximation Guarantee
• Theorem 2: Strategy Weight Convergence
• Theorem 3: Time and Space Complexity
• Definition 4: Submodular Function
• Theorem 5: Diversity is Submodular
• Theorem 6: Weighted Combination
• Corollary 7: MODE's Score is Submodular
• Theorem 8: Main Approximation Theorem
• Theorem 9: Weight Convergence
• Theorem 10: Time Complexity