Federated Online Prediction from Experts with Differential Privacy: Separations and Regret Speed-ups

TL;DR

This work studies differentially private federated online prediction from experts (OPE) across stochastic and oblivious adversaries, introducing Fed-DP-OPE-Stoch that attains a $1/\sqrt{m}$ per-client regret speed-up under DP with logarithmic communication. It establishes fundamental lower bounds for oblivious adversaries, showing limited or no improvement from client collaboration in general, and introduces Fed-SVT to achieve an $m$-fold speed-up in the realizable setting, with near-optimal guarantees up to logarithmic factors. The approach blends local gradient estimation, tree-based private aggregation (DP-FW), and sparse-vector techniques to achieve communication-efficient privacy-preserving federation. Theoretical results are complemented by numerical experiments on synthetic data and MovieLens, confirming the practical benefits of the proposed algorithms. Overall, the paper advances the DP federated online learning literature by separating stochastic and oblivious regimes and by offering provably fast, privacy-preserving federated OPE methods with empirical validation.

Abstract

We study the problems of differentially private federated online prediction from experts against both stochastic adversaries and oblivious adversaries. We aim to minimize the average regret on $m$ clients working in parallel over time horizon $T$ with explicit differential privacy (DP) guarantees. With stochastic adversaries, we propose a Fed-DP-OPE-Stoch algorithm that achieves $\sqrt{m}$-fold speed-up of the per-client regret compared to the single-player counterparts under both pure DP and approximate DP constraints, while maintaining logarithmic communication costs. With oblivious adversaries, we establish non-trivial lower bounds indicating that collaboration among clients does not lead to regret speed-up with general oblivious adversaries. We then consider a special case of the oblivious adversaries setting, where there exists a low-loss expert. We design a new algorithm Fed-SVT and show that it achieves an $m$-fold regret speed-up under both pure DP and approximate DP constraints over the single-player counterparts. Our lower bound indicates that Fed-SVT is nearly optimal up to logarithmic factors. Experiments demonstrate the effectiveness of our proposed algorithms. To the best of our knowledge, this is the first work examining the differentially private online prediction from experts in the federated setting.

Figures (6)

• Figure 1: Per-client regret.
• Figure 2: Per-client regret.
• Figure 3: Binary tree with depth $j = 2$ in \ref{['alg:frank-wolfe']}.
• Figure 4: Comparison between Fed-DP-OPE-Stoch and Limited Updates with different random seeds.
• Figure 5: Comparison between Fed-SVT and Sparse-Vector with different random seeds.

Theorems & Definitions (57)

• Definition 1: $(\varepsilon,\delta)$-DP
• Remark 1: Comparison with asi2022private
• Theorem 1
• proof : Proof sketch of DP guarantees and communication costs
• proof : Proof sketch of regret upper bound
• Corollary 1
• Remark 2
• Remark 3
• Theorem 2
• Remark 4