Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

S-BDT: Distributed Differentially Private Boosted Decision Trees

Thorsten Peinemann, Moritz Kirschte, Joshua Stock, Carlos Cotrini, Esfandiar Mohammadi

TL;DR

S-BDT tackles the challenge of protecting individual training points in distributed gradient boosted decision trees without sacrificing utility. It achieves tighter privacy-utility guarantees via a combination of subsampling, leaf-balanced non-spherical Gaussian noise, and an individual Rényi filter that reuses data across training rounds, including non-IID streams, while supporting distributed collaboration. The authors derive tight, per-leaf and subsampling Rényi DP bounds and demonstrate substantial privacy budget savings (e.g., >$50\%$ on Abalone and >$30\%$ on Adult/Spambase) with comparable RMSE/AUC to state-of-the-art methods. Empirically, S-BDT also shows improvements in streaming non-IID settings and scales to distributed learning, making it a practical DP framework for privacy-preserving GBDTs in real-world, sensitive-data environments.

Abstract

We introduce S-BDT: a novel $(\varepsilon,δ)$-differentially private distributed gradient boosted decision tree (GBDT) learner that improves the protection of single training data points (privacy) while achieving meaningful learning goals, such as accuracy or regression error (utility). S-BDT uses less noise by relying on non-spherical multivariate Gaussian noise, for which we show tight subsampling bounds for privacy amplification and incorporate that into a Rényi filter for individual privacy accounting. We experimentally reach the same utility while saving $50\%$ in terms of epsilon for $\varepsilon \le 0.5$ on the Abalone regression dataset (dataset size $\approx 4K$), saving $30\%$ in terms of epsilon for $\varepsilon \le 0.08$ for the Adult classification dataset (dataset size $\approx 50K$), and saving $30\%$ in terms of epsilon for $\varepsilon\leq0.03$ for the Spambase classification dataset (dataset size $\approx 5K$). Moreover, we show that for situations where a GBDT is learning a stream of data that originates from different subpopulations (non-IID), S-BDT improves the saving of epsilon even further.

S-BDT: Distributed Differentially Private Boosted Decision Trees

TL;DR

S-BDT tackles the challenge of protecting individual training points in distributed gradient boosted decision trees without sacrificing utility. It achieves tighter privacy-utility guarantees via a combination of subsampling, leaf-balanced non-spherical Gaussian noise, and an individual Rényi filter that reuses data across training rounds, including non-IID streams, while supporting distributed collaboration. The authors derive tight, per-leaf and subsampling Rényi DP bounds and demonstrate substantial privacy budget savings (e.g., > on Abalone and > on Adult/Spambase) with comparable RMSE/AUC to state-of-the-art methods. Empirically, S-BDT also shows improvements in streaming non-IID settings and scales to distributed learning, making it a practical DP framework for privacy-preserving GBDTs in real-world, sensitive-data environments.

Abstract

We introduce S-BDT: a novel -differentially private distributed gradient boosted decision tree (GBDT) learner that improves the protection of single training data points (privacy) while achieving meaningful learning goals, such as accuracy or regression error (utility). S-BDT uses less noise by relying on non-spherical multivariate Gaussian noise, for which we show tight subsampling bounds for privacy amplification and incorporate that into a Rényi filter for individual privacy accounting. We experimentally reach the same utility while saving in terms of epsilon for on the Abalone regression dataset (dataset size ), saving in terms of epsilon for for the Adult classification dataset (dataset size ), and saving in terms of epsilon for for the Spambase classification dataset (dataset size ). Moreover, we show that for situations where a GBDT is learning a stream of data that originates from different subpopulations (non-IID), S-BDT improves the saving of epsilon even further.
Paper Structure (40 sections, 18 theorems, 28 equations, 6 figures, 3 tables, 7 algorithms)

This paper contains 40 sections, 18 theorems, 28 equations, 6 figures, 3 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Overview
  3. Preliminaries
  4. Differential Privacy
  5. Gradient Boosted Decision Trees
  6. Individual Rényi filter
  7. Subsampled Rényi differential privacy
  8. Problem statement & related work
  9. Random split selection
  10. Leaf Noising
  11. Distributed learning: Batched Updates
  12. Discussion of closely related work
  13. Tight Individual RDP for subsampled non-spherical multivariate Noise
  14. Exact RDP bounds for Poisson subsampling
  15. Individual RDP bound for Leaf-balanced noise
  16. ...and 25 more sections

Key Result

Theorem 5

Let $\alpha$ and $\rho(\alpha)$ be fixed, $\mathcal{X}$ be a dataset space, and $M$ be a sequence of adaptively chosen mechanisms $M_i: \Pi_{j=1}^{i-1} \mathcal{R}_{j} \times \mathcal{X} \mapsto \mathcal{R}_i$ for $i \in {\left\{\,1,\dots,k\,\right\}}\xspace$, i.e. $M_i$ has the outputs of all previ

Figures (6)

  • Figure 1: Schematic overview of $\text{S-BDT}$ ($\varepsilon=0.5$) classifying two-dimensional concentric circles where the inner yellow circle arrives after the (here: 700) regular training rounds. The numbers ➀ to ➄ refer to $\text{S-BDT}$'s key technical features (cf. \ref{['sec:overview']}).
  • Figure 2: Individual Rényi filter (IRF) boosts the average privacy leakage (dotted line) closer to the worst-case accounted one. The noise scale is calibrated on the regular tree training rounds, but data points that did not consume their accounted budget are used for free in extra rounds (here: 100 rounds). On the abalone dataset, we 1) measure for every data point (x-axis) the privacy leakage by how much the gradient sum of $\text{S-BDT}$'s leaf changes after removing one data point and 2) aggregate it across the ensemble (y-axis).
  • Figure 3: Comparison of utility-privacy tradeoff of our $\text{S-BDT}$ and the SOTA by Maddock et al. Maddock_2022. Regression error (RMSE) (Abalone) and AUC (Adult and Spambase) of 200 runs (for Spambase and $\varepsilon\leq0.1$: 1000 runs) vs. privacy budget $\varepsilon$ ($(b)$ and $(c)$ in log-scale). The transparent area is the standard error.
  • Figure 4: Ablation studies of our improvements: Regression error (RMSE) and AUC of 200 runs. The transparent area is the standard error. For Abalone we set $\varepsilon=0.105$, number of trees $T_\text{regular}=150$, depth $d=2$, subsampling ratio $\gamma=0.1$, leaf-balanced noise parameter $r_1=0.2$, privacy budget ratio for initial score $\varepsilon_\text{init}/\varepsilon=0.1$, $\varepsilon_\text{ds}=0.005$ and clipping bound $g^*=0.1$. For Adult, we set $\varepsilon=0.02$, $T_\text{regular}=200$, $d=5$, $\gamma=0.005$, $r_1=0.1$, $g^*=0.5, h^*=0.1$. A smaller $r_1$ value means more privacy budget for Hessian sum compared to gradient sum. $r_1=0.5$ deactivates leaf-balanced noise, $\gamma=1.0$ subsampling and $\varepsilon_\text{init}/\varepsilon=0.0$ the initial score.
  • Figure 5: Learning a stream of non-IID data: Regression error (RMSE) (Abalone) and AUC (Adult) of 200 runs vs. privacy budget $\varepsilon$. The transparent area is the standard error.
  • ...and 1 more figures

Theorems & Definitions (44)

  • Definition 1: Neighboring datasets
  • Definition 2: DP, DBLP:conf/tcc/DworkMNS06
  • Definition 3: Rényi Divergence, renyi_divergence
  • Definition 4: Rényi DP, Definition 4 in DBLP:journals/corr/Mironov17
  • Theorem 5: Adaptive sequential composition for RDP, feldman2020individual Theorem 3.1
  • Corollary 6: RDP to DP, Thm. 21 in balle2019hypothesis or Prop. 12 in canonne2020discrete
  • Definition 7: $L_2$ sensitivity
  • Theorem 8: Gaussian mechanism, privacybookDBLP:journals/corr/Mironov17
  • Theorem 9: Post-Processing DBLP:journals/corr/Mironov17
  • Definition 10: Individual Rényi Differential Privacy
  • ...and 34 more