Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Immersion and Invariance-based Coding for Privacy-Preserving Federated Learning

Haleh Hayati, Carlos Murguia, Nathan van de Wouw

TL;DR

A privacy-preserving FL framework that combines differential privacy and system immersion tools from control theory is introduced that can be tailored to offer any desired level of differential privacy for both local and global model parameters, while maintaining the same accuracy and convergence rate as standard FL algorithms.

Abstract

Federated learning (FL) has emerged as a method to preserve privacy in collaborative distributed learning. In FL, clients train AI models directly on their devices rather than sharing data with a centralized server, which can pose privacy risks. However, it has been shown that despite FL's partial protection of local data privacy, information about clients' data can still be inferred from shared model updates during training. In recent years, several privacy-preserving approaches have been developed to mitigate this privacy leakage in FL, though they often provide privacy at the cost of model performance or system efficiency. Balancing these trade-offs presents a significant challenge in implementing FL schemes. In this manuscript, we introduce a privacy-preserving FL framework that combines differential privacy and system immersion tools from control theory. The core idea is to treat the optimization algorithms used in standard FL schemes (e.g., gradient-based algorithms) as a dynamical system that we seek to immerse into a higher-dimensional system (referred to as the target optimization algorithm). The target algorithm's dynamics are designed such that, first, the model parameters of the original algorithm are immersed in its parameters; second, it operates on distorted parameters; and third, it converges to an encoded version of the true model parameters from the original algorithm. These encoded parameters can then be decoded at the server to retrieve the original model parameters. We demonstrate that the proposed privacy-preserving scheme can be tailored to offer any desired level of differential privacy for both local and global model parameters, while maintaining the same accuracy and convergence rate as standard FL algorithms.

Immersion and Invariance-based Coding for Privacy-Preserving Federated Learning

TL;DR

A privacy-preserving FL framework that combines differential privacy and system immersion tools from control theory is introduced that can be tailored to offer any desired level of differential privacy for both local and global model parameters, while maintaining the same accuracy and convergence rate as standard FL algorithms.

Abstract

Federated learning (FL) has emerged as a method to preserve privacy in collaborative distributed learning. In FL, clients train AI models directly on their devices rather than sharing data with a centralized server, which can pose privacy risks. However, it has been shown that despite FL's partial protection of local data privacy, information about clients' data can still be inferred from shared model updates during training. In recent years, several privacy-preserving approaches have been developed to mitigate this privacy leakage in FL, though they often provide privacy at the cost of model performance or system efficiency. Balancing these trade-offs presents a significant challenge in implementing FL schemes. In this manuscript, we introduce a privacy-preserving FL framework that combines differential privacy and system immersion tools from control theory. The core idea is to treat the optimization algorithms used in standard FL schemes (e.g., gradient-based algorithms) as a dynamical system that we seek to immerse into a higher-dimensional system (referred to as the target optimization algorithm). The target algorithm's dynamics are designed such that, first, the model parameters of the original algorithm are immersed in its parameters; second, it operates on distorted parameters; and third, it converges to an encoded version of the true model parameters from the original algorithm. These encoded parameters can then be decoded at the server to retrieve the original model parameters. We demonstrate that the proposed privacy-preserving scheme can be tailored to offer any desired level of differential privacy for both local and global model parameters, while maintaining the same accuracy and convergence rate as standard FL algorithms.
Paper Structure (21 sections, 4 theorems, 52 equations, 5 figures, 3 tables, 2 algorithms)

This paper contains 21 sections, 4 theorems, 52 equations, 5 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Standard Federated Learning
  4. Privacy Requirements
  5. Privacy-preserving FL Problem
  6. Immersion-based Privacy-preserving FL: Secure Aggregation
  7. Immersion-based Coding for Privacy- Preserving FL
  8. Summary SIFL Algorithm solving Problem \ref{['problem1']}: Privacy-Preserving Aggregation
  9. Global model privacy in FL
  10. Affine Solution to Problem \ref{['problem2']}
  11. Extended SIFL Algorithm solving Problems \ref{['problem1']} and \ref{['problem2']}: Privacy-Preserving Aggregation and Broadcasting
  12. Privacy Guarantees
  13. Differential Privacy
  14. Immersion-based Coding Differential Privacy Guarantee
  15. Solution to Problem \ref{['problem3']}
  16. ...and 6 more sections

Key Result

Proposition 1

(Solution to Problem problem1) For given full rank matrix $\Pi_1 \in \mathbb{R}^{\tilde{n} \times n}$, matrix $N_1 \in \mathbb{R}^{\tilde{n} \times (\tilde{n} - n)}$ expanding the kernel of $\Pi_{1}^{L}$(i.e., $\Pi_{1}^{L}N_1=\mathbf{0}$), and random process $r_1^t \in \mathbb{R}^{(\tilde{n} - n)}$, target optimizer: and inverse function: provide a solution to Problem problem1.

Figures (5)

  • Figure 1: Flowchart of the extended SIFL method.
  • Figure 2: The comparison of the accuracy of FL network in each iteration with and without the proposed privacy mechanism.
  • Figure 3: The comparison of the accuracy and loss of FL with and without privacy for the Fashion-MNIST database.
  • Figure 4: The comparison of the training time of FL with and without the proposed privacy mechanism.
  • Figure 5: The comparison of the accuracy of FL, SIFL M2, and NbAFL for various privacy levels $\epsilon=1,5,10,20$.

Theorems & Definitions (11)

  • Remark 1
  • Proposition 1
  • Proposition 2
  • Remark 2
  • Definition 1: Adjacency Dwork
  • Definition 2: $(\epsilon,\delta)$-Differential Privacy Dwork
  • Definition 3: Sensitivity
  • Theorem 1
  • Theorem 2
  • Remark 3
  • ...and 1 more