Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CatBoost: unbiased boosting with categorical features

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, Andrey Gulin

TL;DR

The paper identifies a fundamental bias in gradient boosting: target leakage induces a prediction shift between training and test distributions. It introduces ordered boosting combined with ordered target statistics to eliminate leakage while efficiently handling high-cardinality categorical features. The CatBoost framework demonstrates state-of-the-art performance against XGBoost and LightGBM across diverse datasets, with ordered TS and mode choices offering outsized gains on small datasets. This work provides practical, leakage-resistant boosting methods and an open-source implementation that improves predictive accuracy on tabular data with categorical features.

Abstract

This paper presents the key algorithmic techniques behind CatBoost, a new gradient boosting toolkit. Their combination leads to CatBoost outperforming other publicly available boosting implementations in terms of quality on a variety of datasets. Two critical algorithmic advances introduced in CatBoost are the implementation of ordered boosting, a permutation-driven alternative to the classic algorithm, and an innovative algorithm for processing categorical features. Both techniques were created to fight a prediction shift caused by a special kind of target leakage present in all currently existing implementations of gradient boosting algorithms. In this paper, we provide a detailed analysis of this problem and demonstrate that proposed algorithms solve it effectively, leading to excellent empirical results.

CatBoost: unbiased boosting with categorical features

TL;DR

The paper identifies a fundamental bias in gradient boosting: target leakage induces a prediction shift between training and test distributions. It introduces ordered boosting combined with ordered target statistics to eliminate leakage while efficiently handling high-cardinality categorical features. The CatBoost framework demonstrates state-of-the-art performance against XGBoost and LightGBM across diverse datasets, with ordered TS and mode choices offering outsized gains on small datasets. This work provides practical, leakage-resistant boosting methods and an open-source implementation that improves predictive accuracy on tabular data with categorical features.

Abstract

This paper presents the key algorithmic techniques behind CatBoost, a new gradient boosting toolkit. Their combination leads to CatBoost outperforming other publicly available boosting implementations in terms of quality on a variety of datasets. Two critical algorithmic advances introduced in CatBoost are the implementation of ordered boosting, a permutation-driven alternative to the classic algorithm, and an innovative algorithm for processing categorical features. Both techniques were created to fight a prediction shift caused by a special kind of target leakage present in all currently existing implementations of gradient boosting algorithms. In this paper, we provide a detailed analysis of this problem and demonstrate that proposed algorithms solve it effectively, leading to excellent empirical results.

Paper Structure

This paper contains 51 sections, 9 theorems, 37 equations, 4 figures, 10 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Categorical features
  4. Related work on categorical features
  5. Target statistics
  6. Greedy TS
  7. Holdout TS
  8. Leave-one-out TS
  9. Ordered TS
  10. Prediction shift and ordered boosting
  11. Prediction shift
  12. Related work on prediction shift
  13. Analysis of prediction shift
  14. Ordered boosting
  15. Ordered boosting with categorical features
  16. ...and 36 more sections

Key Result

Theorem 1

1. If two independent samples $\mathcal{D}_1$ and $\mathcal{D}_2$ of size $n$ are used to estimate $h^1$ and $h^2$, respectively, using Equation eq:gradient_approximation, then $\mathbb{E}_{\mathcal{D}_1, \mathcal{D}_2} F^2(\mathbf{x})=f^*(\mathbf{x})+O(1/2^{n})$ for any $\mathbf{x}\in \{0,1\}^2$. 2

Figures (4)

  • Figure 1: Ordered boosting principle, examples are ordered according to $\sigma$.
  • Figure 2: Relative error of Plain boosting mode compared to Ordered boosting mode depending on the fraction of the dataset.
  • Figure 3: Relative change in logloss for a given allowed complexity compared to the absence of feature combinations.
  • Figure 4: Relative change in logloss for a given number of permutations $s$ compared to $s=1$,

Theorems & Definitions (9)

  • Theorem 1
  • Proposition 1
  • Proposition 2
  • Lemma 1
  • Lemma 2
  • Proposition 3
  • Lemma 3
  • Lemma 4
  • Proposition 4