Privacy-Preserving Model Transcription with Differentially Private Synthetic Distillation

TL;DR

The work tackles privacy leakage in deployed models by introducing DPSD, a data-free, DP-guaranteed method to transcribe a private-data-trained teacher into a privacy-preserving student via a trainable generator. It creates a unified competitive-cooperative framework with three players, enabling synthetic data generation, DP-informed annotation, and adversarial generator updates to closely match private data distribution while protecting data and labels. The authors provide formal privacy proofs for data- and label-sensitive settings and analyze convergence, complemented by extensive experiments across 8 datasets and 26 baselines, showing state-of-the-art performance under various privacy budgets and even in federated settings. The approach offers a practical pathway for deploying high-accuracy models with privacy guarantees, and the results highlight a favorable privacy-utility tradeoff, though high-dimensional data pose ongoing challenges and invite improvements with newer generators.

Abstract

While many deep learning models trained on private datasets have been deployed in various practical tasks, they may pose a privacy leakage risk as attackers could recover informative data or label knowledge from models. In this work, we present \emph{privacy-preserving model transcription}, a data-free model-to-model conversion solution to facilitate model deployment with a privacy guarantee. To this end, we propose a cooperative-competitive learning approach termed \emph{differentially private synthetic distillation} that learns to convert a pretrained model (teacher) into its privacy-preserving counterpart (student) via a trainable generator without access to private data. The learning collaborates with three players in a unified framework and performs alternate optimization: i)~the generator is learned to generate synthetic data, ii)~the teacher and student accept the synthetic data and compute differential private labels by flexible data or label noisy perturbation, and iii)~the student is updated with noisy labels and the generator is updated by taking the student as a discriminator for adversarial training. We theoretically prove that our approach can guarantee differential privacy and convergence. The transcribed student has good performance and privacy protection, while the resulting generator can generate private synthetic data for downstream tasks. Extensive experiments clearly demonstrate that our approach outperforms 26 state-of-the-arts.

Figures (9)

• Figure 1: A model T directly trained on private data may leak privacy in practical deployment, which inspires us to propose model transcription that can convert T into a privacy-preserving model S without access to private data.
• Figure 2: Overview of our differentially private synthetic distillation approach. Left: Our approach can transcribe a pretrained teacher into a privacy-preserving student without available private data via a trainable generator by competitive-cooperative learning. It collaborates with three players in a unified framework, where the student takes part in cooperation with the teacher under differential privacy and competition with the generator. Right: The competitive-cooperative learning freezes the teacher, initializes the student and generator, and then alternately updates the student and generator in each iteration round. First, the generator generates synthetic data from random vectors $\textbf{z}$ under Gaussian distribution. After that, the student predictions $\Phi_s(\textbf{x};\textbf{w}_s)$ and teacher predictions $\Phi_t(\textbf{x};\textbf{w}_t)$ are used to perform differentially private annotation to the synthetic data, which provides switchable data-sensitive or label-sensitive privacy protection with normalization as well as Gaussian mechanism or randomized response mechanism, respectively. Specially, top-$k$ scheme for value masking or index selection is used to reduce the privacy budget and improve model accuracy. Finally, the student and generator are alternately updated by decoupled knowledge distillation with noisy annotations and adversarial learning with the student as a discriminator, respectively.
• Figure 3: Schematic student learning (Top). In existing approaches, learning with public data can avoid privacy issue but suffers from student convergence to the teacher, and learning with private data enables the student to converge to the teacher but leaks privacy. Our approach takes advantage of both sides by learning with synthetic data, where the final student converges to the teacher and the representation distribution of the generated synthetic data finally matches private data, i.e., achieving similar t-SNE representation visualization (Bottom).
• Figure 4: Evaluation on model transcription. We show the test accuracy of transcribed ResNet34 students on ImageNet with different pretrained deep learning models as teachers under a low failure probability of $\delta=10^{-5}$ and different privacy budget $\varepsilon$. We also provide a comparison to the recent DPDFD liu2023model under $\varepsilon=1$, showing better accuracy achieved by our DPSD.
• Figure 5: The effect of the key operations and parameter settings in differentially private annotation.

Theorems & Definitions (19)

• Theorem 1
• Theorem 2
• Definition 1: Differential Privacy (DP) dwork2016jpc
• Definition 2: R$\acute{\textbf{e}}$nyi Differential Privacy (RDP) mironov2017renyi
• Theorem 3: Convert RDP to DP mironov2017renyi
• Theorem 4: Post-processing dwork2014algorithmic
• Theorem 5: Gaussian Mechanism
• Lemma 1
• Proof
• Lemma 2