Privacy-Preserving Student Learning with Differentially Private Data-Free Distillation

TL;DR

This paper presents an effective teacher-student learning approach to train privacy-preserving deep learning models via differentially private data-free distillation, and proposes a label differential privacy algorithm termed selective randomized response to protect the label information.

Abstract

Deep learning models can achieve high inference accuracy by extracting rich knowledge from massive well-annotated data, but may pose the risk of data privacy leakage in practical deployment. In this paper, we present an effective teacher-student learning approach to train privacy-preserving deep learning models via differentially private data-free distillation. The main idea is generating synthetic data to learn a student that can mimic the ability of a teacher well-trained on private data. In the approach, a generator is first pretrained in a data-free manner by incorporating the teacher as a fixed discriminator. With the generator, massive synthetic data can be generated for model training without exposing data privacy. Then, the synthetic data is fed into the teacher to generate private labels. Towards this end, we propose a label differential privacy algorithm termed selective randomized response to protect the label information. Finally, a student is trained on the synthetic data with the supervision of private labels. In this way, both data privacy and label privacy are well protected in a unified framework, leading to privacy-preserving models. Extensive experiments and analysis clearly demonstrate the effectiveness of our approach.

Figures (5)

• Figure 1: The framework of our differentially private data-free distillation approach. It aims to train a privacy-preserving student model $\textbf{S}$ with teacher-student learning. First, a teacher $\textbf{T}$ is well trained on private data and serves as a fixed discriminator to pre-train a generator $\textbf{G}$ in a data-free manner. Then, massive synthetic data is generated from noisy code $\textbf{z}$ with the generator and fed into the teacher and student $\textbf{S}$ to query differentially private labels with selective randomized response. Finally, with the synthetic data and noisy labels, the student is trained by regressing the teacher knowledge. In this way, both the data privacy and label privacy are well protected in a unified framework, leading to a privacy-preserving student model $\textbf{S}$ doing the distillation with final labels and outputs of student. In the selective randomized response, we use the output of the student model combined with a threshold $t$ to reduce the number of possible labels and obtain $I$. We implement $\varepsilon$-DP with return $RR_\varepsilon(I, \mathbf{y}_t)$ if correct label in $I$ and $Uniform(I)$ if correct label not in $I$.
• Figure 2: The effect of different amount of synthetic data ($\varepsilon$=1).
• Figure 3: The effect of different number of stages ($\varepsilon$=10).
• Figure 4: The examples of the generated synthetic data. From top to bottom: MNIST, FMNIST, CIFAR10, CelebA-G and CelebA-H.
• Figure 5: The results of model-inversion attack against the students trained on MNIST with DataLens (top), ALIBI (middle) and DP-DFD (bottom).