On the Inflation of KNN-Shapley Value

TL;DR

This work targets value inflation in Shapley-based data valuation, focusing on KNN-Shapley. It proposes Calibrated KNN-Shapley (CKNN-Shapley), which enforces a minimum training-subset size $|\mathcal{S}|\ge T$ to calibrate the threshold at $0$ and reduce spurious contributions from small subsets. Through extensive experiments on image, text, and other domains, CKNN-Shapley consistently reduces misidentification of detrimental samples and improves data-quality assessment, while preserving efficiency. The authors also demonstrate CKNN-Shapley’s applicability to learning with mislabeled data, online learning with streams, and active labeling, highlighting practical impact for data-centric ML tasks.

Abstract

Shapley value-based data valuation methods, originating from cooperative game theory, quantify the usefulness of each individual sample by considering its contribution to all possible training subsets. Despite their extensive applications, these methods encounter the challenge of value inflation - while samples with negative Shapley values are detrimental, some with positive values can also be harmful. This challenge prompts two fundamental questions: the suitability of zero as a threshold for distinguishing detrimental from beneficial samples and the determination of an appropriate threshold. To address these questions, we focus on KNN-Shapley and propose Calibrated KNN-Shapley (CKNN-Shapley), which calibrates zero as the threshold to distinguish detrimental samples from beneficial ones by mitigating the negative effects of small-sized training subsets. Through extensive experiments, we demonstrate the effectiveness of CKNN-Shapley in alleviating data valuation inflation, detecting detrimental samples, and assessing data quality. We also extend our approach beyond conventional classification settings, applying it to diverse and practical scenarios such as learning with mislabeled data, online learning with stream data, and active learning for label annotation.

Figures (7)

• Figure 1: Illustration of KNN-Shapley value inflation. The bar plot displays the histogram of KNN-Shapley values for training samples in the SST-2 dataset socher2013recursive. For the purpose of visualization, we merge the samples with extremely small or large values into the leftmost or rightmost bars. With a segmentation of 20 equally-sized bins based on the ascending order of their values, the red line illustrates KNN performance with a specific bin removed from the training set, while the dashed green line represents KNN performance on the entire training set. By comparing the red and green lines, the detrimental bin can be identified, as the performance improves upon its removal. While samples with negative KNN-Shapley values are generally detrimental, a notable observation is the green shallow highlighted region, where samples are harmful to the learning task despite having positive KNN-Shapley values, indicating the issue of KNN-Shapley value inflation.
• Figure 2: Comparison of KNN-Shapley and CKNN-Shapley value on the SST-2 dataset socher2013recursive, where each dashed line represents a training sample associated with its KNN-Shapley and CKNN-Shapley values, respectively, and the green region is the misidentified detrimental samples from Figure \ref{['fig:motivation']}. The red dashed lines denote the samples that are incorrectly identified by KNN-Shapley but correctly identified by CKNN-Shapley.
• Figure 3: Execution time and parameter analysis. A shows the execution time by second in the logarithm of three KNN-Shapley-based data valuation approaches; B and C display the classification performance trend of our CKNN-Shapley with different values of $K$ and $T$.
• Figure 4: The classification performance of KNN on datasets MNIST, FMNIST, and CIFAR10 varies with different training sets and flip ratios. The standard KNN utilizes the full training set, including mislabeled data, whereas KNN-Shapley-based methods start by excluding samples having negative Shapley values from the training set, and then apply the KNN classifier.
• Figure 5: In-depth exploration of CKNN-Shapley on CIFAR10 with 0.3 flip ratio. A depicts the sizes of detrimental and mislabelled samples, where $\mathcal{I}$ denotes the set of samples with non-positive Shapley values but not mislabeled, $\mathcal{II}$ presents the set of joint detrimental and mislabeled samples, and $\mathcal{III}$ is the set of mislabeled samples with positive Shapley values. B shows the value distribution of these three sets. C displays the visual examples of detrimental or mislabeled samples and normal samples.