Mislabeled examples detection viewed as probing machine learning models: concepts, survey and extensive benchmark

TL;DR

It is shown that most mislabeled detection methods can be viewed as probing trained machine learning models using a few core principles, and a modular framework is formalized that encompasses these methods, parameterized by only 4 building blocks.

Abstract

Mislabeled examples are ubiquitous in real-world machine learning datasets, advocating the development of techniques for automatic detection. We show that most mislabeled detection methods can be viewed as probing trained machine learning models using a few core principles. We formalize a modular framework that encompasses these methods, parameterized by only 4 building blocks, as well as a Python library that demonstrates that these principles can actually be implemented. The focus is on classifier-agnostic concepts, with an emphasis on adapting methods developed for deep learning models to non-deep classifiers for tabular data. We benchmark existing methods on (artificial) Completely At Random (NCAR) as well as (realistic) Not At Random (NNAR) labeling noise from a variety of tasks with imperfect labeling rules. This benchmark provides new insights as well as limitations of existing methods in this setup.

Figures (26)

• Figure 1: Illustration of a ground truth distribution $\mathbb{P}\left(X,Y\right)$ decomposed as $\mathbb{P}\left(X\right)$ on the y-axis and $\mathbb{P}\left(Y|X\right)$ on the x-axis and a sample of 100 data points from this distribution. In this toy distribution, we distinguish 4 different cases represented as 4 quadrants: (1)$\mathbb{P}\left(X\right)$ is dense, $\mathbb{P}\left(Y|X\right)$ is low entropy: we are pretty confident that the ground truth class is so any example would be mislabeled (2)$\mathbb{P}\left(X\right)$ is dense, $\mathbb{P}\left(Y|X\right)$ is high entropy, we cannot distinguish between classes and thus we would be unable to tell correctly labeled from mislabeled examples (3)$\mathbb{P}\left(X\right)$ is scattered, but $\mathbb{P}\left(Y|X\right)$ is low entropy so we can assume that the ground truth class is and any example should be deemed mislabeled (4) Since $\mathbb{P}\left(X\right)$ is scattered, it is more difficult to detect that $\mathbb{P}\left(Y|X\right)$ is high entropy by looking at the data only, it is likely that mislabeled detection methods would fail in the absence of further assumptions.
• Figure 2: Data pipeline for different learning strategies when in the presence of labeling noise, where an intermediary step uses a detection method to assign trust scores to every example, then splits the dataset into a trusted and an untrusted part.
• Figure 3: Whereas machine learning consists in choosing a model that best fits the data (from left to right), model-probing mislabeled detection methods follow the opposite direction and probe a trained model in order to give diagnostics on a set of examples (from right to left).
• Figure 4: Schematic summary of model-probing detection methods, with their 4 components.
• Figure 5: This code reads "consider a gradient boosted tree model (GBM) as a progressive ensemble, compute margins for all examples at every iteration during training, sum them up to obtain scalar trust scores".

Theorems & Definitions (1)

• Definition 2.1: Trust score