Distributionally Robust Coreset Selection under Covariate Shift

TL;DR

The paper tackles coreset selection when deployment distributions are uncertain due to covariate shift. It introduces Distributionally Robust Coreset Selection (DRCS), derives a computable upper bound on the worst-case weighted validation error under distributional deviation, and then greedily selects coresets to minimize this bound. The approach is developed for convex training with strong convexity and extended to kernelized and NTK-inspired deep learning contexts, with theoretical guarantees and empirical validation on tabular and image tasks. DRCS offers practical data-efficiency benefits by reducing training data while maintaining robust performance across plausible future distributions.

Abstract

Coreset selection, which involves selecting a small subset from an existing training dataset, is an approach to reducing training data, and various approaches have been proposed for this method. In practical situations where these methods are employed, it is often the case that the data distributions differ between the development phase and the deployment phase, with the latter being unknown. Thus, it is challenging to select an effective subset of training data that performs well across all deployment scenarios. We therefore propose Distributionally Robust Coreset Selection (DRCS). DRCS theoretically derives an estimate of the upper bound for the worst-case test error, assuming that the future covariate distribution may deviate within a defined range from the training distribution. Furthermore, by selecting instances in a way that suppresses the estimate of the upper bound for the worst-case test error, DRCS achieves distributionally robust training instance selection. This study is primarily applicable to convex training computation, but we demonstrate that it can also be applied to deep learning under appropriate approximations. In this paper, we focus on covariate shift, a type of data distribution shift, and demonstrate the effectiveness of DRCS through experiments.

Figures (10)

• Figure 1: The concept of coreset selection in this study. In the left panel, each plot shows the distribution of the training data, where each column represents patterns of distribution changes, while each row represents patterns of instance selection. Let gray lines represent the learned results with specified weights and all instances, while green lines the retrained results with specified weights and selected instances. The goal is to obtain a selection pattern that can suppress the degradation of test error, even in the worst-case distribution for each selection pattern. In this figure, through the three distributions, Selection 1 can be considered a better choice than Selection 2. It is practically impossible since we need to explore such worst-case test error for all distributions and selection patterns.
• Figure 2: The concept of coreset selection in this study. This figures also show the distribution of the training data. The green area indicates the bound of model parameters obtained when retraining is performed, and we can analyticaly calculate it before retraining. We perform coreset selection to minimize the worst-case test error in the distribution, where the bound becomes the largest among all possible distributions.
• Figure 3: This figure illustrates an upper bound of the validation error in this study. Both figures show the distribution of the validation data. We calculate a bound of model parameters and use this to derive an upper bound of the validation error. In this figure, the blue and red areas represent that the validation data in these areas have a determined classification. On the other hand, the validation data in the green area does not have a determined classification. As a result, the upper bound of the validation error can be reached in cases where all instances in the green area are incorrectly classified. Since the bound of the model parameters depends on how to select instances such as (a) and (b), we perform coreset selection in a distribution where the upper bound of the validation error is minimized.
• Figure 6: We compare our proposed method with several instance selection baselines with respect to the weighted validation accuracy ($1-{\rm VaEr}$). Our method exhibits superior performance.
• Figure 9: The results represent the model performance across varying values of lambda. The top row corresponds to the weighted validation accuracy ($1-{\rm VaEr}$), and the bottom row to the lower bound of the worst-case weighted validation accuracy ($1-{\rm WrVaEr^{\rm UB}}$). The first column shows results for $\lambda = n \cdot 10^{-3}$, the second column for $\lambda = \lambda_{\rm best}$ by cross-varidation, the third column for $\lambda = n \cdot 10^{-1.5}$, and the fourth column for $\lambda = n$.

Theorems & Definitions (26)

• Definition 3.1
• Definition 3.2
• Theorem 3.3
• Lemma A.1
• proof
• Lemma A.2
• proof
• Corollary A.3
• Lemma A.4
• proof