Learning Privacy-Preserving Student Networks via Discriminative-Generative Distillation

TL;DR

This work proposes a discriminative-generative distillation approach to learn privacy-preserving deep models, taking models as bridge to distill knowledge from private data and then transfer it to learn a student network via two streams.

Abstract

While deep models have proved successful in learning rich knowledge from massive well-annotated data, they may pose a privacy leakage risk in practical deployment. It is necessary to find an effective trade-off between high utility and strong privacy. In this work, we propose a discriminative-generative distillation approach to learn privacy-preserving deep models. Our key idea is taking models as bridge to distill knowledge from private data and then transfer it to learn a student network via two streams. First, discriminative stream trains a baseline classifier on private data and an ensemble of teachers on multiple disjoint private subsets, respectively. Then, generative stream takes the classifier as a fixed discriminator and trains a generator in a data-free manner. After that, the generator is used to generate massive synthetic data which are further applied to train a variational autoencoder (VAE). Among these synthetic data, a few of them are fed into the teacher ensemble to query labels via differentially private aggregation, while most of them are embedded to the trained VAE for reconstructing synthetic data. Finally, a semi-supervised student learning is performed to simultaneously handle two tasks: knowledge transfer from the teachers with distillation on few privately labeled synthetic data, and knowledge enhancement with tangent-normal adversarial regularization on many triples of reconstructed synthetic data. In this way, our approach can control query cost over private data and mitigate accuracy degradation in a unified manner, leading to a privacy-preserving student model. Extensive experiments and analysis clearly show the effectiveness of the proposed approach.

Figures (9)

• Figure 1: The models that are learned with direct access to private data may leak privacy. To address that, we aim to learn privacy-preserving models by knowledge transfer from directly-trained discriminative model(s), combined with knowledge enhancement using synthetic data generated by generative model.
• Figure 2: Overview of the approach. The privacy-preserving student learning is performed with discriminative-generative distillation via two streams. First, discriminative stream trains a baseline classifier on private data and an ensemble of multiple teachers on disjoint private subsets, and generative stream takes the baseline as a fixed discriminator to train a generator in a data-free manner. The generator is employed to generate massive synthetic data that are used to pretrain a variational autoencoder (VAE). Then, the synthetic data are splitted into two parts: a few of them are fed into teacher ensemble in discriminative stream to query labels by noisy aggregation, and most of them are embedded into the VAE space and reconstructed with and without latent code perturbation. Finally, two streams converge to perform semi-supervised student learning by transferring teacher knowledge with few labeled synthetic data and regularizing with massive VAE-reconstructed synthetic data.
• Figure 3: Model accuracy during training and privacy cost under different queries
• Figure 4: The effect of different generators on student accuracy (%). Here, RAW means learning with private data.
• Figure 5: The generated images with different generator learning approaches. It is obvious that the examples generated by data-free learned generator protect privacy better.

Theorems & Definitions (2)

• Theorem 1
• Proof