Towards Few-Call Model Stealing via Active Self-Paced Knowledge Distillation and Diffusion-Based Image Generation

TL;DR

The paper tackles black-box model stealing under a strict query budget by synthesizing a proxy data set with diffusion models and distilling the teacher's knowledge into a student via an active self-paced learning framework (ASPKD). It introduces three core components: diffusion-based proxy data generation, an active learning step that selects informative samples for labeling within the API call limit, and a self-paced labeling strategy for remaining data using nearest-neighbor pseudo-labeling in the student latent space. Empirical results on CIFAR-10, Food-101, and FER+ show that ASPKD outperforms four state-of-the-art baselines across multiple datasets and architectures, with notable gains in very few-call regimes. The work underscores security risks in MLaaS and provides a public implementation to facilitate further defense research and rapid benchmarking.

Abstract

Diffusion models showcase strong capabilities in image synthesis, being used in many computer vision tasks with great success. To this end, we propose to explore a new use case, namely to copy black-box classification models without having access to the original training data, the architecture, and the weights of the model, i.e. the model is only exposed through an inference API. More specifically, we can only observe the (soft or hard) labels for some image samples passed as input to the model. Furthermore, we consider an additional constraint limiting the number of model calls, mostly focusing our research on few-call model stealing. In order to solve the model extraction task given the applied restrictions, we propose the following framework. As training data, we create a synthetic data set (called proxy data set) by leveraging the ability of diffusion models to generate realistic and diverse images. Given a maximum number of allowed API calls, we pass the respective number of samples through the black-box model to collect labels. Finally, we distill the knowledge of the black-box teacher (attacked model) into a student model (copy of the attacked model), harnessing both labeled and unlabeled data generated by the diffusion model. We employ a novel active self-paced learning framework to make the most of the proxy data during distillation. Our empirical results on three data sets confirm the superiority of our framework over four state-of-the-art methods in the few-call model extraction scenario. We release our code for free non-commercial use at https://github.com/vladhondru25/model-stealing.

Figures (9)

• Figure 1: The proposed pipeline for model stealing starts by generating proxy images using a diffusion model. Then, proxy images are gradually annotated by the black-box teacher model and further used to train the student model via an active learning scheme. At the same time, the remaining proxy images are pseudo-labeled via a nearest neighbor scheme that operates in the latent space of the student. The pseudo-labeled images are also used to optimize the student via a self-paced learning scheme. Best viewed in color.
• Figure 2: Samples of generated images by GLIDE Nichol-ICML-2021b (top row) and Stable Diffusion Rombach-CVPR-2022 (bottom row) for the CIFAR-10 classes.
• Figure 3: Samples of generated images by Stable Diffusion Rombach-CVPR-2022 for some of the Food-101 classes.
• Figure 4: Samples of generated images by SDXL podell-sdxl-2023 for the FER+ classes.
• Figure 5: Real and synthetic CIFAR-10 images plotted using t-SNE. There is a high overlap between the real and proxy data.