Improving Data Quality with Training Dynamics of Gradient Boosting Decision Trees

TL;DR

The paper presents training-dynamics metrics for Gradient Boosting Decision Trees to quantify per-instance learning behavior on tabular data, enabling detection of mislabeled or hard examples. It introduces two data-cleaning strategies: a simple threshold on the product $m_j = c_j \cdot \mu_j$ and an iterative per-instance weighting scheme $w_j$ that down-weights suspicious examples over $E$ rounds. Evaluations on synthetic (NCAR/NNAR) and real datasets (Breast Cancer, Adult), plus an industry forbidden-items case, show that the weighting approach often yields superior noise-detection and higher downstream metrics (F1, PRAUC) after cleaning. The work demonstrates practical applicability by deploying a GBDT-based pipeline that simultaneously reasons about data quality and model training, with potential extensions to automatic relabeling and class-conditional mislabeling scenarios.

Abstract

Real world datasets contain incorrectly labeled instances that hamper the performance of the model and, in particular, the ability to generalize out of distribution. Also, each example might have different contribution towards learning. This motivates studies to better understanding of the role of data instances with respect to their contribution in good metrics in models. In this paper we propose a method based on metrics computed from training dynamics of Gradient Boosting Decision Trees (GBDTs) to assess the behavior of each training example. We focus on datasets containing mostly tabular or structured data, for which the use of Decision Trees ensembles are still the state-of-the-art in terms of performance. Our methods achieved the best results overall when compared with confident learning, direct heuristics and a robust boosting algorithm. We show results on detecting noisy labels in order clean datasets, improving models' metrics in synthetic and real public datasets, as well as on a industry case in which we deployed a model based on the proposed solution.

Figures (5)

• Figure 1: Dataset cartography illustration based on training dynamics: average confidence, variability and correctness, allowing to map the instances to how the model evolved to estimate the outputs along iterations and classify points into easy, ambiguous, hard and even noisy.
• Figure 2: Correctness versus Confidence for each instance, as well as their multiplication (coded in different colors).
• Figure 3: Examples of violin plot with the distributions of (a) $m_j$ correctness and confidence product, and (b) $w_j$ learned weights across the "binary synthetic" dataset instances
• Figure 4: Synthetic datasets with 10% noise, binary version (top) and multiclass version (bottom)
• Figure 5: Cartography metrics for a forbidden item dataset with 4 classes. The colorbar represents correctness in the range [0,1].