Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated One-Shot Learning with Data Privacy and Objective-Hiding

Maximilian Egger, Rüdiger Urbanke, Rawad Bitar

TL;DR

This work addresses federated learning under dual privacy constraints: protecting clients' private data and concealing the federator's training objective. It introduces a novel end-to-end one-shot protocol that unifies secure aggregation, secret sharing-based MPC, and a graph-based private information retrieval scheme to achieve objective-hiding with information-theoretic privacy. The scheme comprises three stages—task assignment, sharing, and query/reconstruction—leveraging a public unlabeled dataset and additively separable objectives, with explicit rates for the sharing ($R_{ ext{share}}$) and PIR ($R_{ ext{PIR}}$) stages and privacy guarantees against colluding parties. A key contribution is a tailored graph-based PIR scheme designed for arbitrary task incidence, enabling efficient retrieval of aggregated client labels while ensuring data privacy and federator objective privacy; extensions to protect the federator from leaking client results and comparisons to existing PIR frameworks are also provided. The approach enables non-linear function computations via federated knowledge distillation, offering a practical, information-theoretic privacy-preserving pathway for federated one-shot learning in settings with heterogeneous client data and diverse objectives.

Abstract

Privacy in federated learning is crucial, encompassing two key aspects: safeguarding the privacy of clients' data and maintaining the privacy of the federator's objective from the clients. While the first aspect has been extensively studied, the second has received much less attention. We present a novel approach that addresses both concerns simultaneously, drawing inspiration from techniques in knowledge distillation and private information retrieval to provide strong information-theoretic privacy guarantees. Traditional private function computation methods could be used here; however, they are typically limited to linear or polynomial functions. To overcome these constraints, our approach unfolds in three stages. In stage 0, clients perform the necessary computations locally. In stage 1, these results are shared among the clients, and in stage 2, the federator retrieves its desired objective without compromising the privacy of the clients' data. The crux of the method is a carefully designed protocol that combines secret-sharing-based multi-party computation and a graph-based private information retrieval scheme. We show that our method outperforms existing tools from the literature when properly adapted to this setting.

Federated One-Shot Learning with Data Privacy and Objective-Hiding

TL;DR

This work addresses federated learning under dual privacy constraints: protecting clients' private data and concealing the federator's training objective. It introduces a novel end-to-end one-shot protocol that unifies secure aggregation, secret sharing-based MPC, and a graph-based private information retrieval scheme to achieve objective-hiding with information-theoretic privacy. The scheme comprises three stages—task assignment, sharing, and query/reconstruction—leveraging a public unlabeled dataset and additively separable objectives, with explicit rates for the sharing () and PIR () stages and privacy guarantees against colluding parties. A key contribution is a tailored graph-based PIR scheme designed for arbitrary task incidence, enabling efficient retrieval of aggregated client labels while ensuring data privacy and federator objective privacy; extensions to protect the federator from leaking client results and comparisons to existing PIR frameworks are also provided. The approach enables non-linear function computations via federated knowledge distillation, offering a practical, information-theoretic privacy-preserving pathway for federated one-shot learning in settings with heterogeneous client data and diverse objectives.

Abstract

Privacy in federated learning is crucial, encompassing two key aspects: safeguarding the privacy of clients' data and maintaining the privacy of the federator's objective from the clients. While the first aspect has been extensively studied, the second has received much less attention. We present a novel approach that addresses both concerns simultaneously, drawing inspiration from techniques in knowledge distillation and private information retrieval to provide strong information-theoretic privacy guarantees. Traditional private function computation methods could be used here; however, they are typically limited to linear or polynomial functions. To overcome these constraints, our approach unfolds in three stages. In stage 0, clients perform the necessary computations locally. In stage 1, these results are shared among the clients, and in stage 2, the federator retrieves its desired objective without compromising the privacy of the clients' data. The crux of the method is a carefully designed protocol that combines secret-sharing-based multi-party computation and a graph-based private information retrieval scheme. We show that our method outperforms existing tools from the literature when properly adapted to this setting.
Paper Structure (24 sections, 7 theorems, 44 equations, 6 figures, 1 table)

This paper contains 24 sections, 7 theorems, 44 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. PIR
  4. Graph-Based PIR
  5. Secure Aggregation in FL
  6. Private Function Computation and Distributed Computing
  7. Symmetric PIR
  8. Knowledge Distillation in FL
  9. Preliminaries and System Model
  10. Problem Illustration Through the Lens of Fine-tuning Large-Language Models
  11. General Scheme
  12. Task Assignment Stage
  13. Sharing Stage
  14. Query Stage
  15. Reconstruction Stage
  16. ...and 9 more sections

Key Result

Proposition 1

The rate of the proposed sharing scheme is

Figures (6)

  • Figure 1: High-level description of a three-stage protocol that first establishes in two stages (assignment and sharing) an MDS-coded data storage pattern based on secret sharing that encodes the aggregated clients computation results for all objectives, and then queries the result for the objective of interest.
  • Figure 2: Illustration of the sharing and query phase of our protocol. The objective of interest is hidden from the curious clients. The clients do not learn about the other clients' results. The federator only learns the aggregate clients' results for the objective of interest.
  • Figure 3: Communication Cost in $s c$ symbols in $\mathbb{F}_q$ compared to our three-stage protocol paired with GXSTPIR jia2020asymptotic and the Star-Product scheme with optimized storage code dimension $k_\mathcal{C}^\star$ according to \ref{['lemma:optimal_comm']}. The latter is limited to $\rho = n$, i.e., to non-graph-based settings. The parameters are chosen as $n=10$, $T=10$, and $z_s=z_q=1$.
  • Figure 4: Communication Cost in $s c$ symbols in $\mathbb{F}_q$ compared to our three-stage protocol paired with GXSTPIR jia2020asymptotic and the Star-Product scheme with optimized storage code dimension $k_\mathcal{C}^\star$ according to \ref{['lemma:optimal_comm']}. The latter is limited to $\rho = n$, i.e., to non-graph-based settings. The parameters are chosen as $n=100$, $T=20$, and $z_s=z_q=5$.
  • Figure 5: Secret Sharing rates compared to GXSTPIR jia2020asymptotic and the Star-Product scheme with optimized storage code dimension $k_\mathcal{C}^\star$ according to \ref{['lemma:optimal_comm']}. The latter is limited to $\rho = n$, i.e., to non-graph-based settings. The parameters are chosen as $n=10$, $T=10$, and $z_s=z_q=1$.
  • ...and 1 more figures

Theorems & Definitions (20)

  • Remark 1
  • Definition 1: Data Privacy from Clients
  • Definition 2: Objective-Hiding
  • Definition 3: Data Privacy from Federator
  • Example 1
  • Remark 2
  • Proposition 1: Sharing Rate
  • proof
  • Theorem 1: PIR Rate
  • proof
  • ...and 10 more