Unified Privacy Guarantees for Decentralized Learning via Matrix Factorization

TL;DR

This paper addresses privacy guarantees in decentralized learning by extending the Matrix Factorization (MF) privacy mechanism from centralized DP-SGD to DP-DL. It generalizes MF to handle rectangular and rank-deficient workloads and multiple trust models, representing attacker knowledge as a linear MF with A = B C to obtain Gaussian Differential Privacy guarantees even under adaptive gradients. A novel algorithm, Mafalda-SGD, is proposed to optimize correlated noise locally, achieving improved privacy-utility trade-offs. Empirical results on synthetic and real-world graphs show tighter PNDP accounting and superior performance of Mafalda-SGD relative to baselines, underscoring the practical impact of principled MF-based DP-DL design.

Abstract

Decentralized Learning (DL) enables users to collaboratively train models without sharing raw data by iteratively averaging local updates with neighbors in a network graph. This setting is increasingly popular for its scalability and its ability to keep data local under user control. Strong privacy guarantees in DL are typically achieved through Differential Privacy (DP), with results showing that DL can even amplify privacy by disseminating noise across peer-to-peer communications. Yet in practice, the observed privacy-utility trade-off often appears worse than in centralized training, which may be due to limitations in current DP accounting methods for DL. In this paper, we show that recent advances in centralized DP accounting based on Matrix Factorization (MF) for analyzing temporal noise correlations can also be leveraged in DL. By generalizing existing MF results, we show how to cast both standard DL algorithms and common trust models into a unified formulation. This yields tighter privacy accounting for existing DP-DL algorithms and provides a principled way to develop new ones. To demonstrate the approach, we introduce MAFALDA-SGD, a gossip-based DL algorithm with user-level correlated noise that outperforms existing methods on synthetic and real-world graphs.

Figures (10)

• Figure 1: Overview of the MF-D-SGD algorithm written as a single equation.
• Figure 2: PNDP loss accounting for DP-D-SGD: comparison between the original accounting of cyffersMuffliatoPeertoPeerPrivacy2022 (dashed lines) and our accounting (solid lines) under the PNDP trust model. We report the Rényi divergence (fixing $\alpha = 2$ w.l.o.g., since the results are proportional to $\alpha$) for the minimum, maximum, and average in error bars over nodes at a given distance.
• Figure 3: Comparison of Mafalda-SGD with the non-private baseline, AntiPGD, and standard DP-SGD under local differential privacy on the Housing dataset with the Facebook Ego graph. Left: test loss over time (AntiPGD does not appear because the test loss is always greater than $6$), averaged over $20$ runs. Right: final test loss as a function of the LDP privacy budget, averaged over $20$ runs. In both cases, data points are distributed uniformly at random across nodes.
• Figure : Florentine Families
• Figure : (a) Ego, $\mu=0.5$

Theorems & Definitions (14)

• Definition 1: Gaussian Differential Privacy (GDP) --- Dong22GDPpillutla2025correlatednoisemechanismsdifferentially
• Definition 2: Gossip matrix
• Definition 3: Linear DL algorithm
• Definition 4: Attacker knowledge
• Theorem 5
• Definition 6: Sensitivity under participation, generalized
• Theorem 7
• Remark 8
• Definition 9
• Lemma 10