Histogram-Based Federated XGBoost using Minimal Variance Sampling for Federated Tabular Data

TL;DR

Federated learning for tabular data is challenging, and this work proposes histogram-based Federated XGBoost (F-XGB) that leverages Minimal Variance Sampling (MVS) to select informative training examples. Using regularized gradients $\hat{g}_i=\sqrt{g_i^2+\lambda h_i^2}$, the method biases sampling toward low-variance, informative instances within a histogram-based, horizontal FL framework. Across the FedTab benchmark, F-XGB with MVS consistently improves accuracy and RMSE over no- and uniform-sampling variants and is competitive with, and sometimes superior to, centralized XGBoost. The FedTab dataset collection provides a practical benchmark for future FL studies, highlighting that sampling in FL can yield substantial performance gains for tabular data.

Abstract

Federated Learning (FL) has gained considerable traction, yet, for tabular data, FL has received less attention. Most FL research has focused on Neural Networks while Tree-Based Models (TBMs) such as XGBoost have historically performed better on tabular data. It has been shown that subsampling of training data when building trees can improve performance but it is an open problem whether such subsampling can improve performance in FL. In this paper, we evaluate a histogram-based federated XGBoost that uses Minimal Variance Sampling (MVS). We demonstrate the underlying algorithm and show that our model using MVS can improve performance in terms of accuracy and regression error in a federated setting. In our evaluation, our model using MVS performs better than uniform (random) sampling and no sampling at all. It achieves both outstanding local and global performance on a new set of federated tabular datasets. Federated XGBoost using MVS also outperforms centralized XGBoost in half of the studied cases.

Figures (6)

• Figure 1: Tabular data, each row is a unique observation and the columns indicate features. Values can be numerical and categorical.
• Figure 2: General architecture of XGBoost. XGBoost combines predictions of several individual models, called "weak learners", into final prediction.
• Figure 3: System overview of federated XGBoost using sampling.
• Figure 4: Top 1% evaluation accuracy for different F-XGB sampling methods. 50% sampling fraction is used for F-XGB on all datasets and mean scores across 5 runs
• Figure 5: Change in top 1% accuracy for local client test set vs. global evaluation set on different datasets. Scores are averaged over 5 runs.