Bilateral Differentially Private Vertical Federated Boosted Decision Trees

TL;DR

MaskedXGBoost tackles privacy-preserving vertical federated XGBoost by introducing bilateral differential privacy through information noising. PP generates structured noises aligned with the splitting matrix M and AP perturbs gradients and Hessians with these noises, enabling secure joint split evaluation with reduced reliance on heavy cryptography. Theoretical analysis provides utility concentration and bilateral DP guarantees for both active and passive parties, while extensive experiments on six datasets demonstrate strong utility, faster training times, and robust empirical privacy against label and attribute inference attacks. The approach offers a practical balance between privacy and efficiency for real-world federated tree ensembles, with potential extensions to broader privacy-preserving ML tasks.

Abstract

Federated learning is a distributed machine learning paradigm that enables collaborative training across multiple parties while ensuring data privacy. Gradient Boosting Decision Trees (GBDT), such as XGBoost, have gained popularity due to their high performance and strong interpretability. Therefore, there has been a growing interest in adapting XGBoost for use in federated settings via cryptographic techniques. However, it should be noted that these approaches may not always provide rigorous theoretical privacy guarantees, and they often come with a high computational cost in terms of time and space requirements. In this paper, we propose a variant of vertical federated XGBoost with bilateral differential privacy guarantee: MaskedXGBoost. We build well-calibrated noise to perturb the intermediate information to protect privacy. The noise is structured with part of its ingredients in the null space of the arithmetical operation for splitting score evaluation in XGBoost, helping us achieve consistently better utility than other perturbation methods and relatively lower overhead than encryption-based techniques. We provide theoretical utility analysis and empirically verify privacy preservation. Compared with other algorithms, our algorithm's superiority in both utility and efficiency has been validated on multiple datasets.

Figures (15)

• Figure 1: An example of XGBoost and splitting vector. The final prediction for instance is the sum of the predictions from each regression tree. At the first level of Tree1, we split the instances based on whether the individual is female or not, which generates the splitting vector $[1,0,1,0]$.
• Figure 2: Federated XGBoost problem setting.
• Figure 3: MaskedXGBoost overview. MaskedXGBoost is composed of four steps: initialization, noise calibration, information noising, and joint optimal splitting candidate searching. AP and PP first determine the necessary algorithm parameters in the initialization step. In the noise calibration step, PP builds well-calibrated noise according to the categorical matrix which ensures the consistently better property of MaskedXGBoost. In the information noising step, AP uses the noises PP sent to perturb the sensitive information $g$ and $h$, then AP sends the information back to PP. Finally, AP and PP jointly find the optimal splitting candidate. Then, splitting continues from the newly constructed nodes until it reaches the maximum depth of the tree. AP and PP stand for active party and passive party respectively.
• Figure 4: Utility of different methods for different privacy budgets.
• Figure 5: Training process of the four XGBoost algorithms with different privacy budgets on Credit 1 dataset

Theorems & Definitions (4)

• Definition 1: Differential Privacy dwork2014algorithmic
• Lemma 1: Sequential Composition dwork2014algorithmic
• Lemma 2: Parallel Composition dwork2014algorithmic
• Definition 2: Local Differential Privacy erlingsson2014rappor