Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Secure Stateful Aggregation: A Practical Protocol with Applications in Differentially-Private Federated Learning

Marshall Ball, James Bell-Clark, Adria Gascon, Peter Kairouz, Sewoong Oh, Zhiye Xie

TL;DR

It is observed that secure stateful aggregation suffices for realizing DP-FTRL-based private federated learning: improving DPFL utility guarantees over the state of the art while maintaining privacy with an untrusted central party.

Abstract

Recent advances in differentially private federated learning (DPFL) algorithms have found that using correlated noise across the rounds of federated learning (DP-FTRL) yields provably and empirically better accuracy than using independent noise (DP-SGD). While DP-SGD is well-suited to federated learning with a single untrusted central server using lightweight secure aggregation protocols, secure aggregation is not conducive to implementing modern DP-FTRL techniques without assuming a trusted central server. DP-FTRL based approaches have already seen widespread deployment in industry, albeit with a trusted central curator who provides and applies the correlated noise. To realize a fully private, single untrusted server DP-FTRL federated learning protocol, we introduce secure stateful aggregation: a simple append-only data structure that allows for the private storage of aggregate values and reading linear functions of the aggregates. Assuming Ring Learning with Errors, we provide a lightweight and scalable realization of this protocol for high-dimensional data in a new security/resource model, Federated MPC : where a powerful persistent server interacts with weak, ephemeral clients. We observe that secure stateful aggregation suffices for realizing DP-FTRL-based private federated learning: improving DPFL utility guarantees over the state of the art while maintaining privacy with an untrusted central party. Our approach has minimal overhead relative to existing techniques which do not yield comparable utility. The secure stateful aggregation primitive and the federated MPC paradigm may be of interest for other practical applications.

Secure Stateful Aggregation: A Practical Protocol with Applications in Differentially-Private Federated Learning

TL;DR

It is observed that secure stateful aggregation suffices for realizing DP-FTRL-based private federated learning: improving DPFL utility guarantees over the state of the art while maintaining privacy with an untrusted central party.

Abstract

Recent advances in differentially private federated learning (DPFL) algorithms have found that using correlated noise across the rounds of federated learning (DP-FTRL) yields provably and empirically better accuracy than using independent noise (DP-SGD). While DP-SGD is well-suited to federated learning with a single untrusted central server using lightweight secure aggregation protocols, secure aggregation is not conducive to implementing modern DP-FTRL techniques without assuming a trusted central server. DP-FTRL based approaches have already seen widespread deployment in industry, albeit with a trusted central curator who provides and applies the correlated noise. To realize a fully private, single untrusted server DP-FTRL federated learning protocol, we introduce secure stateful aggregation: a simple append-only data structure that allows for the private storage of aggregate values and reading linear functions of the aggregates. Assuming Ring Learning with Errors, we provide a lightweight and scalable realization of this protocol for high-dimensional data in a new security/resource model, Federated MPC : where a powerful persistent server interacts with weak, ephemeral clients. We observe that secure stateful aggregation suffices for realizing DP-FTRL-based private federated learning: improving DPFL utility guarantees over the state of the art while maintaining privacy with an untrusted central party. Our approach has minimal overhead relative to existing techniques which do not yield comparable utility. The secure stateful aggregation primitive and the federated MPC paradigm may be of interest for other practical applications.

Paper Structure

This paper contains 33 sections, 5 theorems, 36 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Differentially-private federated learning framework.
  3. Learning with independent noise (differentially-private stochastic gradient descent) and secure aggregation.
  4. A new paradigm for private learning: correlated noise (differentially-private follow-the-regularized-leader).
  5. Our Results
  6. Secure Stateful Aggregation.
  7. The $(\gamma,\beta)$-secure federated MPC paradigm.
  8. A lightweight protocol from RLWE.
  9. Application 1: Releasing private partial sums (introduction to correlated-noise mechanisms).
  10. Application 2: Differentially-private federated learning via DP-FTRL.
  11. Limitations and Future Directions
  12. Stateful Aggregation
  13. Long running aggregation.
  14. Applications of Stateful Aggregation in Distributed Differential Privacy
  15. Baselines
  16. ...and 18 more sections

Key Result

Theorem 1

Assuming that a semi-honest PPT adversary corrupts at most an $\gamma$-fraction of any user cohort, in addition to the server, The protocol given in Figures fig:server_no_dropout, fig:client_no_dropout securely implements the functionality in Figure fig:ideal.

Figures (6)

  • Figure 1: The Stateful Aggregation Functionality
  • Figure 2: Server: No dropout resilience
  • Figure 3: Client $j$ in cohort $i$: No dropout resilience
  • Figure 4: Server: With dropout resilience
  • Figure 5: Client $j$ in cohort $i$: With dropout resilience
  • ...and 1 more figures

Theorems & Definitions (17)

  • Theorem 1
  • Remark 1
  • Remark 2
  • Definition 1: Decisional LWE assumption
  • Definition 2: LWE encryption
  • Definition 3: Additive secret sharing
  • Definition 4: Threshold secret sharing
  • Definition 5: MLWE C:KLSS22
  • Definition 6: Hint-MLWE C:KLSS22
  • Definition 7: Smoothing parameter 1366257
  • ...and 7 more