Data-Free Knowledge Distillation for Deep Neural Networks

TL;DR

The paper tackles the challenge of compressing large neural networks when the original training data is unavailable. It introduces data-free knowledge distillation, leveraging activation-based metadata (top-layer, all-layer, and spectral records) to reconstruct surrogate training data and train a smaller student model. Through experiments on MNIST and CelebA, it demonstrates that richer activation records, especially spectral methods, substantially improve data-free distillation performance, approaching traditional distillation under certain conditions. The work highlights practical benefits for distributing compressed models when data handling or privacy concerns prohibit data sharing, and it calls for standardized metadata formats to support such techniques.

Abstract

Recent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy. However, all of these approaches rely on access to the original training set, which might not always be possible if the network to be compressed was trained on a very large dataset, or on a dataset whose release poses privacy or safety concerns as may be the case for biometrics tasks. We present a method for data-free knowledge distillation, which is able to compress deep neural networks trained on large-scale datasets to a fraction of their size leveraging only some extra metadata to be provided with a pretrained model release. We also explore different kinds of metadata that can be used with our method, and discuss tradeoffs involved in using each of them.

Figures (4)

• Figure 1: A production pipeline for Deep Learning models: an over-parameterized model is trained to high accuracies using the computation power in a data center, then is compressed for deployment in a smartphone.
• Figure 2: The effect of scaling non-linearities to some temperature: the softened activations should provide the student model with more information about how the teacher model generalizes.
• Figure 3: The proposed model compression pipeline: a model is trained in a datacenter and released along with some metadata. Then, another entity uses that metadata to reconstruct a dataset, which is then used to compress the model with Knowledge Distillation. Finally, the model is deployed in a smartphone.
• Figure 4: Overview of the different activation records and methods used to reconstruct the original dataset and train the student network. In (\ref{['pure_distill']}), the student network is trained directly on examples from the original dataset as input, and the teacher's temperature-scaled activations as labels. In (\ref{['top_layer_diagram']}), we keep activation statistics for the top layer of the teacher network. Then, we sample from those, and optimize the input to the teacher to recreate those activations. That reconstructed input is then used to train the student network. (\ref{['all_layers_diagram']}) is very similar to (\ref{['top_layer_diagram']}), but it involves recording statistics, sampling, and recreating activations for all layers of the network. In (\ref{['spectral_all_layers']}), the optimization objective is to reconstruct the entire activation of the network to correspond to a compressed version of the original network activation. This is intended to better capture inter-layer dynamics and is initially done by expanding the activation into a graph Fourier basis and only retaining a fraction of the spectrum coefficients. In order to compute such an expansion more quickly, we consider applying the same method to each pair of layers separately (\ref{['spectral_pair_layers']}). This is less computationally expensive to compute but requires storing eigenvalues for each pair of layers, which is ultimately less space-efficient.