Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient-less Federated Gradient Boosting Trees with Learnable Learning Rates

Chenyang Ma, Xinchi Qiu, Daniel J. Beutel, Nicholas D. Lane

TL;DR

This work develops an innovative framework for horizontal federated XGBoost which does not depend on the sharing of gradients and simultaneously boosts privacy and communication efficiency by making the learning rates of the aggregated tree ensembles learnable.

Abstract

The privacy-sensitive nature of decentralized datasets and the robustness of eXtreme Gradient Boosting (XGBoost) on tabular data raise the needs to train XGBoost in the context of federated learning (FL). Existing works on federated XGBoost in the horizontal setting rely on the sharing of gradients, which induce per-node level communication frequency and serious privacy concerns. To alleviate these problems, we develop an innovative framework for horizontal federated XGBoost which does not depend on the sharing of gradients and simultaneously boosts privacy and communication efficiency by making the learning rates of the aggregated tree ensembles learnable. We conduct extensive evaluations on various classification and regression datasets, showing our approach achieves performance comparable to the state-of-the-art method and effectively improves communication efficiency by lowering both communication rounds and communication overhead by factors ranging from 25x to 700x. Project Page: https://flower.ai/blog/2023-04-19-xgboost-with-flower/

Gradient-less Federated Gradient Boosting Trees with Learnable Learning Rates

TL;DR

This work develops an innovative framework for horizontal federated XGBoost which does not depend on the sharing of gradients and simultaneously boosts privacy and communication efficiency by making the learning rates of the aggregated tree ensembles learnable.

Abstract

The privacy-sensitive nature of decentralized datasets and the robustness of eXtreme Gradient Boosting (XGBoost) on tabular data raise the needs to train XGBoost in the context of federated learning (FL). Existing works on federated XGBoost in the horizontal setting rely on the sharing of gradients, which induce per-node level communication frequency and serious privacy concerns. To alleviate these problems, we develop an innovative framework for horizontal federated XGBoost which does not depend on the sharing of gradients and simultaneously boosts privacy and communication efficiency by making the learning rates of the aggregated tree ensembles learnable. We conduct extensive evaluations on various classification and regression datasets, showing our approach achieves performance comparable to the state-of-the-art method and effectively improves communication efficiency by lowering both communication rounds and communication overhead by factors ranging from 25x to 700x. Project Page: https://flower.ai/blog/2023-04-19-xgboost-with-flower/
Paper Structure (12 sections, 6 equations, 4 figures, 5 tables, 1 algorithm)

This paper contains 12 sections, 6 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Method
  4. Intuitions
  5. Tree Ensembles Aggregation
  6. Learnable Learning Rates by One-layer 1D CNN
  7. FedXGBllr
  8. Experiments
  9. Experiment Setup and Implementations
  10. Experimental Results
  11. Ablation Studies and Analysis
  12. Conclusion and Future Works

Figures (4)

  • Figure 1: Horizontal vs. vertical federated XGBoost.
  • Figure 2: An example of the impact of local data heterogeneity on the performance of XGBoost model.
  • Figure 3: The aggregated tree ensemble. The final prediction given by the weighted sum of all trees.
  • Figure 4: The pipeline. (a) tree ensembles aggregation and (b) one-layer 1D CNN to study the learning rates and output the final prediction result.