Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Principled Federated Random Forests for Heterogeneous Data

Rémi Khellaf, Erwan Scornet, Aurélien Bellet, Julie Josse

TL;DR

This work tackles federated learning for nonparametric CART/Random Forests on horizontally partitioned data, where gradient-based optimization is not available. FedForest uses federated quantile sketching to generate candidate splits and additive sufficient statistics to evaluate impurity reductions exactly from aggregated client summaries, enabling faithful replication of centralized CART. Importantly, it supports splits on the client indicator $H$, providing a nonparametric form of personalization under outcome heterogeneity. Empirical results show FedForest achieves centralized-like predictive performance with communication-efficient training, across synthetic and real heterogeneous datasets, outperforming several federated baselines and parametric models in non-i.i.d. settings.

Abstract

Random Forests (RF) are among the most powerful and widely used predictive models for centralized tabular data, yet few methods exist to adapt them to the federated learning setting. Unlike most federated learning approaches, the piecewise-constant nature of RF prevents exact gradient-based optimization. As a result, existing federated RF implementations rely on unprincipled heuristics: for instance, aggregating decision trees trained independently on clients fails to optimize the global impurity criterion, even under simple distribution shifts. We propose FedForest, a new federated RF algorithm for horizontally partitioned data that naturally accommodates diverse forms of client data heterogeneity, from covariate shift to more complex outcome shift mechanisms. We prove that our splitting procedure, based on aggregating carefully chosen client statistics, closely approximates the split selected by a centralized algorithm. Moreover, FedForest allows splits on client indicators, enabling a non-parametric form of personalization that is absent from prior federated random forest methods. Empirically, we demonstrate that the resulting federated forests closely match centralized performance across heterogeneous benchmarks while remaining communication-efficient.

Principled Federated Random Forests for Heterogeneous Data

TL;DR

This work tackles federated learning for nonparametric CART/Random Forests on horizontally partitioned data, where gradient-based optimization is not available. FedForest uses federated quantile sketching to generate candidate splits and additive sufficient statistics to evaluate impurity reductions exactly from aggregated client summaries, enabling faithful replication of centralized CART. Importantly, it supports splits on the client indicator , providing a nonparametric form of personalization under outcome heterogeneity. Empirical results show FedForest achieves centralized-like predictive performance with communication-efficient training, across synthetic and real heterogeneous datasets, outperforming several federated baselines and parametric models in non-i.i.d. settings.

Abstract

Random Forests (RF) are among the most powerful and widely used predictive models for centralized tabular data, yet few methods exist to adapt them to the federated learning setting. Unlike most federated learning approaches, the piecewise-constant nature of RF prevents exact gradient-based optimization. As a result, existing federated RF implementations rely on unprincipled heuristics: for instance, aggregating decision trees trained independently on clients fails to optimize the global impurity criterion, even under simple distribution shifts. We propose FedForest, a new federated RF algorithm for horizontally partitioned data that naturally accommodates diverse forms of client data heterogeneity, from covariate shift to more complex outcome shift mechanisms. We prove that our splitting procedure, based on aggregating carefully chosen client statistics, closely approximates the split selected by a centralized algorithm. Moreover, FedForest allows splits on client indicators, enabling a non-parametric form of personalization that is absent from prior federated random forest methods. Empirically, we demonstrate that the resulting federated forests closely match centralized performance across heterogeneous benchmarks while remaining communication-efficient.
Paper Structure (29 sections, 5 theorems, 49 equations, 7 figures, 3 tables, 1 algorithm)

This paper contains 29 sections, 5 theorems, 49 equations, 7 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Setting and Background
  3. Centralized CART/RF Split Selection
  4. Heterogeneity Scenarios
  5. Federated CART
  6. Candidate Split Generation
  7. Federated Impurity Reduction
  8. Federated Forests
  9. Bootstrap and Client Subsampling
  10. Parallel tree growth and batched rounds
  11. FedForest Training Protocol
  12. Experiments
  13. Synthetic Data
  14. Real Data
  15. Conclusion
  16. ...and 14 more sections

Key Result

Theorem 3.1

Let $F_\nu^{(j)}(x)=\sum_{k=1}^K \frac{n_{\nu,k}}{n_\nu}F_{\nu,k}^{(j)}(x)$ be the pooled empirical CDF for feature $j$ at node $\nu$, and let $\tilde{F}_\nu^{(j)}$ be obtained by linearly interpolating between the $B$ reported quantile points per client and mixing as above. Then,

Figures (7)

  • Figure 1: Graphical models of the heterogeneity regimes. Dashed arrows indicate possible dependencies between the client variable $H$ and the covariates and/or outcome mechanism, depending on the regime.
  • Figure 2: Methods comparison on homogeneous clients
  • Figure 3: Methods comparison under covariate shift
  • Figure 4: Methods comparison under outcome shift
  • Figure 5: Methods comparison on FedHeart-Disease dataset
  • ...and 2 more figures

Theorems & Definitions (10)

  • Theorem 3.1: Uniform rank error of the reconstructed CDF
  • Corollary 3.2: Approximation of centralized midpoint splits
  • Theorem 3.3: General impurity decomposition
  • Theorem 3.4: AvgImp approximation error on homogeneous data
  • proof
  • proof
  • proof
  • proof : Proof of \ref{['cor:epsilon_approx']}
  • Proposition 1.1: Screening consistency under i.i.d. clients
  • proof