Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guarding the Middle: Protecting Intermediate Representations in Federated Split Learning

Obaidullah Zaland, Sajib Mistry, Monowar Bhuyan

TL;DR

This work proposes k-anonymous differentially private UFSL (KD-UFSL), which leverages privacy-enhancing techniques such as microaggregation and differential privacy to minimize data leakage from the smashed data transferred to the server.

Abstract

Big data scenarios, where massive, heterogeneous datasets are distributed across clients, demand scalable, privacy-preserving learning methods. Federated learning (FL) enables decentralized training of machine learning (ML) models across clients without data centralization. Decentralized training, however, introduces a computational burden on client devices. U-shaped federated split learning (UFSL) offloads a fraction of the client computation to the server while keeping both data and labels on the clients' side. However, the intermediate representations (i.e., smashed data) shared by clients with the server are prone to exposing clients' private data. To reduce exposure of client data through intermediate data representations, this work proposes k-anonymous differentially private UFSL (KD-UFSL), which leverages privacy-enhancing techniques such as microaggregation and differential privacy to minimize data leakage from the smashed data transferred to the server. We first demonstrate that an adversary can access private client data from intermediate representations via a data-reconstruction attack, and then present a privacy-enhancing solution, KD-UFSL, to mitigate this risk. Our experiments indicate that, alongside increasing the mean squared error between the actual and reconstructed images by up to 50% in some cases, KD-UFSL also decreases the structural similarity between them by up to 40% on four benchmarking datasets. More importantly, KD-UFSL improves privacy while preserving the utility of the global model. This highlights its suitability for large-scale big data applications where privacy and utility must be balanced.

Guarding the Middle: Protecting Intermediate Representations in Federated Split Learning

TL;DR

This work proposes k-anonymous differentially private UFSL (KD-UFSL), which leverages privacy-enhancing techniques such as microaggregation and differential privacy to minimize data leakage from the smashed data transferred to the server.

Abstract

Big data scenarios, where massive, heterogeneous datasets are distributed across clients, demand scalable, privacy-preserving learning methods. Federated learning (FL) enables decentralized training of machine learning (ML) models across clients without data centralization. Decentralized training, however, introduces a computational burden on client devices. U-shaped federated split learning (UFSL) offloads a fraction of the client computation to the server while keeping both data and labels on the clients' side. However, the intermediate representations (i.e., smashed data) shared by clients with the server are prone to exposing clients' private data. To reduce exposure of client data through intermediate data representations, this work proposes k-anonymous differentially private UFSL (KD-UFSL), which leverages privacy-enhancing techniques such as microaggregation and differential privacy to minimize data leakage from the smashed data transferred to the server. We first demonstrate that an adversary can access private client data from intermediate representations via a data-reconstruction attack, and then present a privacy-enhancing solution, KD-UFSL, to mitigate this risk. Our experiments indicate that, alongside increasing the mean squared error between the actual and reconstructed images by up to 50% in some cases, KD-UFSL also decreases the structural similarity between them by up to 40% on four benchmarking datasets. More importantly, KD-UFSL improves privacy while preserving the utility of the global model. This highlights its suitability for large-scale big data applications where privacy and utility must be balanced.
Paper Structure (27 sections, 13 equations, 8 figures, 4 tables, 2 algorithms)

This paper contains 27 sections, 13 equations, 8 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Differential Privacy
  5. Definition 1 ($\epsilon$-DP) dwork2014algorithmic:
  6. Definition 2 ($(\epsilon, \delta)$-DP) eddp:
  7. Defintion 3($(\epsilon, \delta)$-DP eddp3) with Normal Distribution):
  8. KD-UFSL: K-Anonymized Differentially Private U-shaped Federated Learning
  9. Threat Model
  10. Privacy Framework
  11. Differential Privacy (DP)
  12. K-Annonymity
  13. Implementation Details
  14. Datasets
  15. Model Training and Hyperparameters
  16. ...and 12 more sections

Figures (8)

  • Figure 1: The clients' data on the top, and reconstructed data from the client's smashed data at the bottom.
  • Figure 2: The process of training UFSL with model-level k-anonymity or micro-aggregation.
  • Figure 3: ResNet-18 architecture split into head, body, and tail networks alongside the inversion network. RB1 refers to ResBlock1 and so on.
  • Figure 4: Actual images and reconstructed images from the client's smashed data of different training strategies, using the inversion network. Each row represents a single model.
  • Figure 5: Accuracy over global rounds of the final global model for KD-UFSL and baselines. CN refers to ConvNet, and RN18 refers to ResNet18.
  • ...and 3 more figures