Accelerating Differentially Private Federated Learning via Adaptive Extrapolation

TL;DR

The paper tackles the challenge of accelerating differentially private federated learning under data heterogeneity without adding hyperparameters or increasing client-side cost. It introduces DP-FedEXP, with LDP-FedEXP and CDP-FedEXP variants, which adaptively set the global step size based on local-update diversity while preserving DP guarantees. The authors derive step-size rules for Gaussian and PrivUnit mechanisms, provide rigorous privacy and non-convex convergence analyses, and demonstrate via synthetic and MNIST experiments that DP-FedEXP achieves faster convergence and higher utility than DP-FedAvg and DP-SCAFFOLD. The work offers a practical, hyperparameter-free approach to DP-FL that remains compatible with other FL acceleration techniques and is particularly beneficial when client resources are constrained.

Abstract

The federated learning (FL) framework enables multiple clients to collaboratively train machine learning models without sharing their raw data, but it remains vulnerable to privacy attacks. One promising approach is to incorporate differential privacy (DP)-a formal notion of privacy-into the FL framework. DP-FedAvg is one of the most popular algorithms for DP-FL, but it is known to suffer from the slow convergence in the presence of heterogeneity among clients' data. Most of the existing methods to accelerate DP-FL require 1) additional hyperparameters or 2) additional computational cost for clients, which is not desirable since 1) hyperparameter tuning is computationally expensive and data-dependent choice of hyperparameters raises the risk of privacy leakage, and 2) clients are often resource-constrained. To address this issue, we propose DP-FedEXP, which adaptively selects the global step size based on the diversity of the local updates without requiring any additional hyperparameters or client computational cost. We show that DP-FedEXP provably accelerates the convergence of DP-FedAvg and it empirically outperforms existing methods tailored for DP-FL.

Figures (3)

• Figure 1: The distance to the optimal solution for the synthetic dataset (left) and test accuracy for the MNIST dataset (right). In both LDP and CDP cases, DP-FedEXP consistently outperforms baseline algorithms.
• Figure 2: The adaptive step size $\eta_g^{(0)}$ at initialization in the LDP setting. Our proposed step size is close to the target step size $\eta^{(0)}_{\mathrm{target}}$ for large $M$ while the naive step size $\tilde{\eta}_g^{(0)}$ is extremely large due to the bias in the numerator and the error does not decrease as $M$ increases.
• Figure 3: Global step sizes for the synthetic dataset (left) and the MNIST dataset (right).

Theorems & Definitions (25)

• Definition 2.1: Central Differential Privacy dwork2014algorithmic
• Definition 2.2: Local Differential Privacy kasiviswanathan2011can
• Remark 3.1: Adaptivity to the Noise Scale
• Proposition 4.1: LDP case
• Proposition 4.2: CDP case
• Theorem 4.5: LDP case
• Theorem 4.6: CDP case
• Lemma 1.1: Gaussian tail bound
• proof
• Lemma 1.2: Tail bound for norm of Gaussian