Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective

TL;DR

The paper tackles the risk of knowledge distillation from proprietary LLMs by logit distributions, introducing a defense grounded in conditional mutual information $\mathcal{I}(X;Z|Y)$. It learns a post-processing transformation $Z' = \mathcal{M}Z$ that provably reduces distillation-relevant information, and optimizes $\mathcal{M}$ via two IB-inspired losses: a cross-entropy surrogate to preserve $\mathcal{I}(Z;Y)$ and a gradient-based term to suppress $\mathcal{I}(X;Z)$ through distillation signals, with a low-rank parameterization $\mathcal{M}=\mathbf{E}+\mathbf{AB}$ for scalability. Experiments across Qwen and Llama models show the method preserves teacher accuracy while significantly degrading distillation performance across multiple datasets (GSM8K, MMLU, MATH) and KD algorithms, including KD, RKLD, AlphaNet, and ABKD. This approach provides a principled, practical mechanism to protect LLM IPs from logit-based KD and lays groundwork for more robust defenses in real-world deployment. The method leverages $Z'$, $Y$, and $X$ in a way that disrupts distillation without compromising user-facing utility, offering a valuable tool for secure model sharing and deployment in API-based ecosystems.

Abstract

Proprietary large language models (LLMs) embody substantial economic value and are generally exposed only as black-box APIs, yet adversaries can still exploit their outputs to extract knowledge via distillation. Existing defenses focus exclusively on text-based distillation, leaving the important logit-based distillation largely unexplored. In this work, we analyze this problem and present an effective solution from an information-theoretic perspective. We characterize distillation-relevant information in teacher outputs using the conditional mutual information (CMI) between teacher logits and input queries conditioned on ground-truth labels. This quantity captures contextual information beneficial for model extraction, motivating us to defend distillation via CMI minimization. Guided by our theoretical analysis, we propose learning a transformation matrix that purifies the original outputs to enhance distillation resistance. We further derive a CMI-inspired anti-distillation objective to optimize this transformation, which effectively removes distillation-relevant information while preserving output utility. Extensive experiments across multiple LLMs and strong distillation algorithms demonstrate that the proposed method significantly degrades distillation performance while preserving task accuracy, effectively protecting models' intellectual property.

Figures (14)

• Figure 1: An illustration of the designed defense algorithm. We propose to learn a transformation matrix that post-processes output logits. By minimizing the conditional mutual information $\mathcal{I}(X;Z|Y)$, we successfully reduce the information beneficial for knowledge distillation within the teacher logits.
• Figure 2: An illustration of the proposed defense framework. As shown in the left subfigure, we introduce a surrogate student model as an adversary and simulate the KD process. We compute the KD loss $\mathcal{L}_\text{total}$ and extract its gradients $\bm{g}$ with respect to the student parameters to quantify the guidance signal carried by the teacher outputs. The right subfigure explains the loss computation used for training the transformation matrix. To diminish context information in teacher logits, we design to minimize the KLD between the transformed logits $Z'$ and the ground-truth label $Y$, while simultaneously maximizing the angular deviation between the gradients $\bm{g}$ and $\bm{g}'$ calculated by the original and transformed logits, respectively.
• Figure 3: Visualization comparison of defensive and original teacher outputs using Qwen2.5-7B on GSM8K
• Figure 4: The loss landscape for Llama-3.1-8B on GSM8K during matrix optimization.
• Figure 5: Accuracy of teacher and distilled student models under varying values of the matrix loss parameter $\lambda$ and decomposition rank $r$.

Theorems & Definitions (13)

• Remark 1
• Theorem 2
• Theorem 3
• Lemma 4
• Remark 5
• Proposition 6
• Theorem 2
• proof
• Theorem 3
• proof