Deep Classifier Mimicry without Data Access

TL;DR

CAKE addresses the challenge of distilling knowledge from a trained teacher without access to the original training data by generating synthetic pairs and diffusing them toward the teacher's decision boundary using a contrastive objective. The method is model-agnostic and incorporates noise and data priors to ensure broad boundary coverage, enabling effective student learning across architectures and scales. Ablation, cross-model, and cross-dataset experiments show CAKE achieves competitive or superior performance relative to data-free baselines while approaching data-dependent baselines with a modest gap. This approach has practical implications for privacy-preserving model updates and cross-domain knowledge transfer, though it also raises concerns about potential misuse and biases in transferable knowledge.

Abstract

Access to pre-trained models has recently emerged as a standard across numerous machine learning domains. Unfortunately, access to the original data the models were trained on may not equally be granted. This makes it tremendously challenging to fine-tune, compress models, adapt continually, or to do any other type of data-driven update. We posit that original data access may however not be required. Specifically, we propose Contrastive Abductive Knowledge Extraction (CAKE), a model-agnostic knowledge distillation procedure that mimics deep classifiers without access to the original data. To this end, CAKE generates pairs of noisy synthetic samples and diffuses them contrastively toward a model's decision boundary. We empirically corroborate CAKE's effectiveness using several benchmark datasets and various architectural choices, paving the way for broad application.

Figures (4)

• Figure 1: Comparison of naive, generative, and CAKE methods for knowledge distillation on the two-moons dataset. The background visualizes teacher (green/purple) and student (blue/red) decision functions, juxtaposed with original data ($\circ$) and synthesized samples ($\triangle$). Naive and generative methods often converge to similar local minima, inducing an ineffective student decision function. In contrast, CAKE generates samples across the entire decision-relevant region, resulting in a student model that accurately learns the data decision function if trained exclusively on its synthetic samples.
• Figure 2: Student model accuracy on CIFAR-10 (y-axis) when trained on synthetic data distilled from ResNet teacher models of different depths. Each group of bars corresponds to a ResNet teacher model of a particular depth (x-axis), and each bar within a group shows the accuracy of the student model distilled from that teacher model, along with its standard deviation as error bars. As desired, CAKE can compress models at a stable accuracy until capacity is too heavily constrained.
• Figure 3: Performance of different student models distilled from teacher models of various model types trained on MNIST: CNNs, MLPs, ResNets, and ViTs (parameter amounts are set to be similar). Each group of bars corresponds to a particular teacher type and each bar within a group shows the accuracy of a particular type of distilled student model, along with its standard deviation as error bars (5 trials). Overall, matching model types consistently provides good results, whereas distillation across types seems to work if the teacher has less inductive bias than the student.
• Figure 4: Synthetic samples generated from a ResNet teacher by CAKE on the MNIST, SVHN, and CIFAR datasets, demonstrating no visual resemblance with original training data.