Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures

TL;DR

Knowledge Distillation (KD) raises IP protection and diversity concerns for LLMs. The paper proposes MoE Expert Signatures, capturing routing patterns of Mixture-of-Experts (expert specialization and collaboration), to detect distillation, and introduces Shadow-MoE to enable black-box detection via proxy MoEs. It formalizes the KD detection task, defines two routing-based signatures, and provides a reproducible benchmark, achieving $>94\%$ accuracy in semi-black-box settings and $100\%$ accuracy in pure black-box settings. The work offers a robust provenance-analysis tool, with implications for IP protection and defense strategies like routing watermarking or randomization.

Abstract

Knowledge Distillation (KD) accelerates training of large language models (LLMs) but poses intellectual property protection and LLM diversity risks. Existing KD detection methods based on self-identity or output similarity can be easily evaded through prompt engineering. We present a KD detection framework effective in both white-box and black-box settings by exploiting an overlooked signal: the transfer of MoE "structural habits", especially internal routing patterns. Our approach analyzes how different experts specialize and collaborate across various inputs, creating distinctive fingerprints that persist through the distillation process. To extend beyond the white-box setup and MoE architectures, we further propose Shadow-MoE, a black-box method that constructs proxy MoE representations via auxiliary distillation to compare these patterns between arbitrary model pairs. We establish a comprehensive, reproducible benchmark that offers diverse distilled checkpoints and an extensible framework to facilitate future research. Extensive experiments demonstrate >94% detection accuracy across various scenarios and strong robustness to prompt-based evasion, outperforming existing baselines while highlighting the structural habits transfer in LLMs.

Figures (3)

• Figure 1: Overview of our method.(a) Problem formulation: detecting whether a suspected student model was distilled from a teacher model, which is challenging when only black-box access is available. (b) Our Shadow-MoE solution: we train proxy Shadow-MoE models to mimic both the suspected teacher or student, then analyze their expert routing patterns through two key measurements, i.e. expert specialization(task-specific activation profiles across different domains) and expert collaboration (co-activation patterns between experts). Similar routing patterns between the shadow models provide evidence of a distillation relationship.
• Figure 2: Predicted scores with the black-box teachers and white-box students setting of Shadow-MoE. We show Wasserstein distances between the teacher's Shadow-MoE proxy and student models for both Expert Specialization (left) and Expert Collaboration (right) metrics. Blue bars represent distilled students, while pink bars represent non-distilled students trained from scratch. Percentage differences indicate the relative reduction in distance for distilled models compared to their non-distilled counterparts. Successfully detected tasks (where distilled models show lower distances than non-distilled) are marked with bold underline. Lower distances indicate stronger routing signature similarity, providing evidence of knowledge distillation.
• Figure 4: Relative Wasserstein distance reduction for distilled models compared to non-distilled models across different training and calibration set combinations. Darker colors indicate larger reductions (stronger detection signals), with percentages showing how much lower the distilled model's distance is relative to the non-distilled model.

Theorems & Definitions (5)

• Definition 2.1: Knowledge Distillation Set
• Definition 2.2: Knowledge Distillation Detection
• Definition 2.3: Shadow-MoE Proxy
• Definition 2.4: Expert Specialization Profile
• Definition 2.5: Expert Collaboration Matrix