PriPHiT: Privacy-Preserving Hierarchical Training of Deep Neural Networks

TL;DR

PriPHiT tackles the challenge of privacy-preserving deep learning training in edge-cloud setups by coupling adversarial early exiting at the edge with differential privacy noise. The two-stage approach—edge pre-training and subsequent edge-cloud training—ensures that sensitive content is suppressed in shared feature maps while task-relevant information is preserved for cloud classifiers. Across facial and medical datasets and multiple architectures, PriPHiT achieves high accuracy on desired tasks while substantially reducing attacker success in inferring sensitive content or reconstructing inputs, even under white-box attacks. The framework is designed for resource-constrained edge devices and offers a tunable privacy-utility trade-off via the privacy budget $ \\epsilon $, making it practically impactful for privacy-aware continual learning in robotics and cloud-assisted systems.

Abstract

The training phase of deep neural networks requires substantial resources and as such is often performed on cloud servers. However, this raises privacy concerns when the training dataset contains sensitive content, e.g., facial or medical images. In this work, we propose a method to perform the training phase of a deep learning model on both an edge device and a cloud server that prevents sensitive content being transmitted to the cloud while retaining the desired information. The proposed privacy-preserving method uses adversarial early exits to suppress the sensitive content at the edge and transmits the task-relevant information to the cloud. This approach incorporates noise addition during the training phase to provide a differential privacy guarantee. We extensively test our method on different facial and medical datasets with diverse attributes using various deep learning architectures, showcasing its outstanding performance. We also demonstrate the effectiveness of privacy preservation through successful defenses against different white-box, deep and GAN-based reconstruction attacks. This approach is designed for resource-constrained edge devices, ensuring minimal memory usage and computational overhead.

Figures (19)

• Figure 1: The proposed method of privacy-preserving hierarchical training and the steps of edge-cloud execution. Top-left, Step 1: The edge feature extractor is adversarially pre-trained with its two early exits using the user's training set images, the desired content labels and the sensitive content labels. Top-right, Step 2-1: The edge is connected to the cloud and the high-performance cloud analyzer is trained using the sent feature maps extracted from the user's training set images and the desired content labels. The adversarial training of the edge also continues using the early exit. Middle-left, Step 2-2: Since the beginning of the edge-cloud connection, the malicious cloud trains a classification attacker and a reconstruction attacker using its own input images and sensitive content labels at the same time as Step 2-1. Middle-right, Step 2-3: Every time the user sends a feature map from the private training set to train an epoch, the malicious cloud performs a simultaneous attack to classify the sensitive content, or to reconstruct the original input. Bottom, Step 3: When the training is finished, the analyzer and the attackers are tested on a new dataset, unseen by all.
• Figure 2: The simultaneous classification attack accuracy during the training phase on the feature maps coming from the user's private data at each epoch for the smiling versus gender experiment on the CelebA dataset using different architectures.
• Figure 3: MSE loss of the simultaneous reconstruction attack during the training phase on the feature maps coming from the user's private data compared with the user's original inputs at each epoch for the smiling versus gender experiment on the CelebA dataset using different architectures.
• Figure 4: The simultaneous attack during the training phase for the smiling versus gender experiment on the FFHQ dataset. Left: The simultaneous classification attack accuracy on the feature maps coming from the user's private data at each epoch. Right: MSE loss of the simultaneous reconstruction attack on the feature maps coming from the user's private data compared with the user's original inputs at each epoch.
• Figure 5: Examples of the results of the deep reconstruction attack on PriPHiT ($\epsilon=1$) in comparison to the similar attacks on the baselines using the unseen inputs from the test subset of the CelebA dataset. Smiling and Gender attributes are selected as the desired and sensitive contents, respectively. Note the suppression of gender indicators while keeping the smiling attribute in the results of the deep reconstruction attacks when the PriPHiT method is used.