Adversarial Sparse Teacher: Defense Against Distillation-Based Model Stealing Attacks Using Adversarial Examples

TL;DR

The paper tackles model stealing via knowledge distillation by introducing Adversarial Sparse Teacher (AST), a defense that trains a teacher to emit sparse, high-entropy logits in response to adversarial inputs. Central to AST is the Exponential Predictive Divergence (EPD) loss, which weights divergences by target probabilities to preserve entropy and mislead potential thieves better than KL-based approaches. Through extensive CIFAR-10 and CIFAR-100 experiments, AST demonstrates strong defensive performance across various teacher–student pairings, often outperforming prior defenses like Nasty and Stingy Teachers, especially with higher-capacity teachers. The method maintains teacher accuracy while significantly reducing the success of distillation-based model stealing, with promising results in data-limited and unlabeled-data scenarios, suggesting practical applicability for IP protection in real-world deployments.

Abstract

We introduce Adversarial Sparse Teacher (AST), a robust defense method against distillation-based model stealing attacks. Our approach trains a teacher model using adversarial examples to produce sparse logit responses and increase the entropy of the output distribution. Typically, a model generates a peak in its output corresponding to its prediction. By leveraging adversarial examples, AST modifies the teacher model's original response, embedding a few altered logits into the output while keeping the primary response slightly higher. Concurrently, all remaining logits are elevated to further increase the output distribution's entropy. All these complex manipulations are performed using an optimization function with our proposed Exponential Predictive Divergence (EPD) loss function. EPD allows us to maintain higher entropy levels compared to traditional KL divergence, effectively confusing attackers. Experiments on CIFAR-10 and CIFAR-100 datasets demonstrate that AST outperforms state-of-the-art methods, providing effective defense against model stealing while preserving high accuracy. The source codes will be made publicly available here soon.

Figures (3)

• Figure 1: Traning scheme of AST. KL loss indicates Kullback-Liebler divergence loss and it uses logits of adversarial and clean samples. CE loss denotes Cross-entropy loss of clean sample probabilities and labels as usual. AST loss is sum of these two loss terms.
• Figure 2: Illustration of the logit responses after the application of softmax temperature. Columns depict the following information: (A) samples of clean images, (B) responses of the baseline model to these clean samples, (C) responses of the baseline model to the adversarial counterparts of the clean images, (D) responses of the NT model to clean samples, (E) responses of our proposed method (AST) to clean images. Below each distribution, the entropy of the distributions are provided.
• Figure 3: The responses of our ResNet18 AST model and AST trained with KL Divergence model to clean samples from two sets of identical classes is displayed. Top row: input images, middle-row:Responses of AST trained with EPD loss, last row: responses of AST trained with KL Divergence loss. Samples in the leftmost 4 columns and righmost 4 columns belong to same classes. The softmax temperature applied to all logits and below each distribution, the entropy value is displayed.