Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Privacy-Preserving Data Deduplication for Enhancing Federated Learning of Language Models (Extended Version)

Aydin Abadi, Vishnu Asutosh Dasu, Sumanta Sarkar

TL;DR

This paper addresses the problem of deduplication in a federated setup by introducing a pioneering protocol, Efficient Privacy-Preserving Multi-Party Deduplication (EP-MPD), which efficiently removes duplicates from multiple clients' datasets without compromising data privacy.

Abstract

Deduplication is a vital preprocessing step that enhances machine learning model performance and saves training time and energy. However, enhancing federated learning through deduplication poses challenges, especially regarding scalability and potential privacy violations if deduplication involves sharing all clients' data. In this paper, we address the problem of deduplication in a federated setup by introducing a pioneering protocol, Efficient Privacy-Preserving Multi-Party Deduplication (EP-MPD). It efficiently removes duplicates from multiple clients' datasets without compromising data privacy. EP-MPD is constructed in a modular fashion, utilizing two novel variants of the Private Set Intersection protocol. Our extensive experiments demonstrate the significant benefits of deduplication in federated learning of large language models. For instance, we observe up to 19.62\% improvement in perplexity and up to 27.95\% reduction in running time while varying the duplication level between 10\% and 30\%. EP-MPD effectively balances privacy and performance in federated learning, making it a valuable solution for large-scale applications.

Privacy-Preserving Data Deduplication for Enhancing Federated Learning of Language Models (Extended Version)

TL;DR

This paper addresses the problem of deduplication in a federated setup by introducing a pioneering protocol, Efficient Privacy-Preserving Multi-Party Deduplication (EP-MPD), which efficiently removes duplicates from multiple clients' datasets without compromising data privacy.

Abstract

Deduplication is a vital preprocessing step that enhances machine learning model performance and saves training time and energy. However, enhancing federated learning through deduplication poses challenges, especially regarding scalability and potential privacy violations if deduplication involves sharing all clients' data. In this paper, we address the problem of deduplication in a federated setup by introducing a pioneering protocol, Efficient Privacy-Preserving Multi-Party Deduplication (EP-MPD). It efficiently removes duplicates from multiple clients' datasets without compromising data privacy. EP-MPD is constructed in a modular fashion, utilizing two novel variants of the Private Set Intersection protocol. Our extensive experiments demonstrate the significant benefits of deduplication in federated learning of large language models. For instance, we observe up to 19.62\% improvement in perplexity and up to 27.95\% reduction in running time while varying the duplication level between 10\% and 30\%. EP-MPD effectively balances privacy and performance in federated learning, making it a valuable solution for large-scale applications.
Paper Structure (76 sections, 4 theorems, 13 equations, 8 figures, 6 tables)

This paper contains 76 sections, 4 theorems, 13 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Overview of Privacy-Preserving Deduplication in $\text{FL}$
  3. Our Contributions
  4. Preliminaries
  5. Notations and Assumptions
  6. Federated Learning ($\text{FL}$)
  7. Causal Language Modeling (CLM)
  8. Security Model
  9. Threat Model Outline
  10. Two-party Computation
  11. Security in the Presence of Passive Adversaries
  12. Pseudorandom Function and Permutation
  13. Oblivious Pseudorandom Function
  14. Trusted Execution Environments
  15. Group PSI ($\text{G-PSI}$)
  16. ...and 61 more sections

Key Result

Theorem 1

Let $f_{\text{G-PSI}\xspace}$ be the functionality defined in Relation equ::func-with-tp. Let $\mathcal{L}$ be the leakage function presented in Definition def::leakage. If $\mathtt{PRP}\xspace$ is a secure pseudorandom permutation, $\text{EG-PSI}^{ (\text{I})}$ (presented in Figure fig:EG-PSI) secu

Figures (8)

  • Figure 1: First Variant of Efficient Group PSI ($\text{EG-PSI}^{ (\text{I})}$).
  • Figure 2: Second Variant of Efficient Group PSI ($\text{EG-PSI}^{ (\text{II})}$).
  • Figure 3: The process of eight clients forming clusters and groups at various levels of a binary tree. At each level, clients belonging to groups $\mathcal{G}\xspace_{ 0}$ and $\mathcal{G}\xspace_{ 1}$ within the same cluster initiate $\text{EG-PSI}$, followed by the updating of their respective local sets.
  • Figure 4: Efficient Privacy-Preserving Multi-Party Deduplication ($\text{EP-MPD}$): the construction of $\text{P-MPD}$.
  • Figure 5: Federated learning with deduplication.
  • ...and 3 more figures

Theorems & Definitions (16)

  • Definition 1
  • Definition 2: Security of $\text{G-PSI}$
  • Definition 3: pair-wise intersection cardinality
  • Definition 4
  • Theorem 1
  • Theorem 2
  • Definition 5: Security of $\text{P-MPD}$
  • Definition 6
  • Theorem 3
  • proof
  • ...and 6 more