Communication-Efficient and Differentially Private Vertical Federated Learning with Zeroth-Order Optimization

TL;DR

This paper tackles privacy and communication bottlenecks in vertical federated learning (VFL) by introducing DPZV, a differential privacy mechanism for zeroth-order VFL that injects calibrated scalar noise on the backward path. By using a two-point zeroth-order gradient estimator and MeZO for memory efficiency, DPZV achieves tunable $(ε,δ)$-DP while maintaining fast convergence, comparable to first-order DP-SGD, under asynchronous updates. Theoretical results establish convergence and DP guarantees, and extensive experiments on four datasets demonstrate superior privacy-utility tradeoffs and reduced communication rounds, highlighting practical impact for privacy-critical, bandwidth-constrained deployments. Overall, DPZV enables efficient, privacy-preserving VFL with adjustable privacy levels and strong empirical performance, paving the way for scalable, secure distributed learning in feature-partitioned environments.

Abstract

Vertical Federated Learning (VFL) enables collaborative model training across feature-partitioned devices, yet its reliance on device-server information exchange introduces significant communication overhead and privacy risks. Downlink communication from the server to devices in VFL exposes gradient-related signals of the global loss that can be leveraged in inference attacks. Existing privacy-preserving VFL approaches that inject differential privacy (DP) noise on the downlink have the natural repercussion of degraded gradient quality, slowed convergence, and excessive communication rounds. In this work, we propose DPZV, a communication-efficient and differentially private ZO-VFL framework with tunable privacy guarantees. Based on zeroth-order (ZO) optimization, DPZV injects calibrated scalar-valued DP noise on the downlink, significantly reducing variance amplification while providing equivalent protection against targeted inference attacks. Through rigorous theoretical analysis, we establish convergence guarantees comparable to first-order DP-SGD, despite relying solely on ZO estimators, and prove that DPZV satisfies $(ε, δ)$-DP. Extensive experiments demonstrate that DPZV consistently achieves a superior privacy-utility tradeoff and requires fewer communication rounds than existing DP-VFL baselines under strict privacy constraints ($ε\leq 10$).

Figures (7)

• Figure 1: Overview of the training procedure in DPZV. Each device perturbs its local model parameters in two random directions to generate a pair of embeddings, which are then transmitted to the server. The server computes the corresponding function evaluations and applies an elementwise difference to approximate the zeroth-order (ZO) gradient. To ensure differential privacy, scalar-valued Gaussian noise is injected into the aggregated ZO estimate. Unlike traditional vector-valued noise in standard DP algorithms, scalar noise is significantly smaller in norm, thereby preserving model utility even under stringent privacy budgets.
• Figure 2: Test Accuracy of VFL Methods on image classification tasks under DP constraints. $\delta$ is set to $1\times 10^{-3}$. DPZV outperforms first-order VFL methods on two datasets and surpasses all other ZO-based methods across all three datasets, showing both a higher accuracy and a faster convergence rate.
• Figure 3: Privacy-Accuracy tradeoff across different datasets and algorithms. We use a constant level of $\delta=1\times 10^{-3}$ and vary $\epsilon$ to simulate different privacy levels. Our algorithm consistently outperforms baselines under tight privacy budget, showing a slower decay in performance than baselines as $\epsilon$ decreases.
• Figure 4: Normalized memory cost in training for each method. DPZV requires the smallest memory allocation in both datasets, almost the same as model memory itself. This shows the memory efficiency of DPZV, allowing superior performance on large-scale neural networks.
• Figure 5: Achieved accuracy under fixed communication cost on CIFAR-10 under different privacy budget. $\delta$ is set to $1\times 10^{-3}$.

Theorems & Definitions (25)

• Definition 3.1: $(\epsilon, \delta)$-Differential Privacy
• Theorem 5.4
• Corollary 5.5
• Lemma 5.6
• Lemma 5.7
• Lemma 5.8
• Lemma 5.9: Model Update With Delay
• Lemma 5.10
• Definition 6.1: Gaussian Differential Privacy
• Theorem 6.5