Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving GBDT Performance on Imbalanced Datasets: An Empirical Study of Class-Balanced Loss Functions

Jiaqi Luo, Yuan Yuan, Shixin Xu

TL;DR

This study tackles the challenge of class imbalance in tabular data by empirically evaluating class-balanced loss functions within Gradient Boosting Decision Tree (GBDT) models. It analyzes three GBDT implementations (XGBoost, LightGBM, SketchBoost) across binary, multi-class, and multi-label tasks, using five losses including two newly introduced asymmetric variants, and assesses performance on 40 datasets with rigorous hyperparameter tuning. Key contributions include a comprehensive benchmark showing that class-balanced losses generally improve GBDT performance on imbalanced data, with multi-label problems showing particularly large gains, and the release of a Python package to facilitate adoption. The results highlight that gains depend on task type and dataset, underscoring the need for careful method-model pairing, and suggest future work in combining these losses with sampling and extending applicability to more tree ensembles.

Abstract

Class imbalance remains a significant challenge in machine learning, particularly for tabular data classification tasks. While Gradient Boosting Decision Trees (GBDT) models have proven highly effective for such tasks, their performance can be compromised when dealing with imbalanced datasets. This paper presents the first comprehensive study on adapting class-balanced loss functions to three GBDT algorithms across various tabular classification tasks, including binary, multi-class, and multi-label classification. We conduct extensive experiments on multiple datasets to evaluate the impact of class-balanced losses on different GBDT models, establishing a valuable benchmark. Our results demonstrate the potential of class-balanced loss functions to enhance GBDT performance on imbalanced datasets, offering a robust approach for practitioners facing class imbalance challenges in real-world applications. Additionally, we introduce a Python package that facilitates the integration of class-balanced loss functions into GBDT workflows, making these advanced techniques accessible to a wider audience.

Improving GBDT Performance on Imbalanced Datasets: An Empirical Study of Class-Balanced Loss Functions

TL;DR

This study tackles the challenge of class imbalance in tabular data by empirically evaluating class-balanced loss functions within Gradient Boosting Decision Tree (GBDT) models. It analyzes three GBDT implementations (XGBoost, LightGBM, SketchBoost) across binary, multi-class, and multi-label tasks, using five losses including two newly introduced asymmetric variants, and assesses performance on 40 datasets with rigorous hyperparameter tuning. Key contributions include a comprehensive benchmark showing that class-balanced losses generally improve GBDT performance on imbalanced data, with multi-label problems showing particularly large gains, and the release of a Python package to facilitate adoption. The results highlight that gains depend on task type and dataset, underscoring the need for careful method-model pairing, and suggest future work in combining these losses with sampling and extending applicability to more tree ensembles.

Abstract

Class imbalance remains a significant challenge in machine learning, particularly for tabular data classification tasks. While Gradient Boosting Decision Trees (GBDT) models have proven highly effective for such tasks, their performance can be compromised when dealing with imbalanced datasets. This paper presents the first comprehensive study on adapting class-balanced loss functions to three GBDT algorithms across various tabular classification tasks, including binary, multi-class, and multi-label classification. We conduct extensive experiments on multiple datasets to evaluate the impact of class-balanced losses on different GBDT models, establishing a valuable benchmark. Our results demonstrate the potential of class-balanced loss functions to enhance GBDT performance on imbalanced datasets, offering a robust approach for practitioners facing class imbalance challenges in real-world applications. Additionally, we introduce a Python package that facilitates the integration of class-balanced loss functions into GBDT workflows, making these advanced techniques accessible to a wider audience.
Paper Structure (23 sections, 10 equations, 3 figures, 6 tables)

This paper contains 23 sections, 10 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. GBDT for imbalanced classification
  4. Class-balanced loss
  5. Methodology
  6. GBDT models
  7. Class-balanced losses for GBDT
  8. Weighted Cross-entropy
  9. Focal Loss
  10. Asymmetric Loss
  11. Asymmetric Cross-entropy
  12. Asymmetric Weighted Cross-entropy
  13. Datasets
  14. Evaluation
  15. Performance
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Binary F1-score absolute improvement.
  • Figure 2: Multi-class F1-score absolute improvement.
  • Figure 3: Multi-label F1-score absolute improvement.