Differentially Private Federated Learning With Time-Adaptive Privacy Spending
Shahrzad Kiani, Nupur Kulkarni, Adam Dziedzic, Stark Draper, Franziska Boenisch
TL;DR
This paper tackles the DP-FL privacy-utility tradeoff by introducing a time-adaptive spend-as-you-go framework that saves privacy budget in early rounds and spends more in later rounds, enabling better learning of fine-grained features without increasing privacy risk. It derives comprehensive privacy accounting using Rényi DP, proves that per-client privacy spending is non-decreasing under the schedule, and shows how optimally permuting saving-based sampling rates can reduce clipping bias. The authors provide theoretical bounds and practical algorithms for selecting when to save vs. spend and how to allocate sampling rates, complemented by experiments on FMNIST, MNIST, Adult, and CIFAR10 that demonstrate improved utility over DP-FedAvg and IDP-FedAvg under heterogeneous budgets. The work has practical implications for deploying DP-FL in real-world settings with diverse privacy constraints, offering a principled approach to allocate privacy budget where it yields the most accuracy gains while maintaining rigorous privacy guarantees.
Abstract
Federated learning (FL) with differential privacy (DP) provides a framework for collaborative machine learning, enabling clients to train a shared model while adhering to strict privacy constraints. The framework allows each client to have an individual privacy guarantee, e.g., by adding different amounts of noise to each client's model updates. One underlying assumption is that all clients spend their privacy budgets uniformly over time (learning rounds). However, it has been shown in the literature that learning in early rounds typically focuses on more coarse-grained features that can be learned at lower signal-to-noise ratios while later rounds learn fine-grained features that benefit from higher signal-to-noise ratios. Building on this intuition, we propose a time-adaptive DP-FL framework that expends the privacy budget non-uniformly across both time and clients. Our framework enables each client to save privacy budget in early rounds so as to be able to spend more in later rounds when additional accuracy is beneficial in learning more fine-grained features. We theoretically prove utility improvements in the case that clients with stricter privacy budgets spend budgets unevenly across rounds, compared to clients with more relaxed budgets, who have sufficient budgets to distribute their spend more evenly. Our practical experiments on standard benchmark datasets support our theoretical results and show that, in practice, our algorithms improve the privacy-utility trade-offs compared to baseline schemes.