Exploiting the Experts: Unauthorized Compression in MoE-LLMs
Pinaki Prasad Guha Neogi, Ahmad Mohammadshirazi, Dheeraj Kulshrestha, Rajiv Ramnath
TL;DR
The paper analyzes a security vulnerability in Mixture-of-Experts LLMs where an adversary can log expert usage, prune nonessential experts, and cheaply fine-tune the retained ones to create a compact, task-specialized surrogate. It introduces an expert-attribution framework to identify critical experts, and demonstrates through experiments on Mixtral and Switch Transformer models that a small set of experts can preserve most task performance, enabling unauthorized compression. It further shows that active learning substantially improves the efficiency of re-alignment, while defense ideas like entangled experts can significantly impede unauthorized compression. The work highlights the dual-use nature of MoE modularity, proposing a structured framework for evaluating prunability-resistance and offering design strategies to harden MoE-LLMs against pruning-based attacks.
Abstract
Mixture-of-Experts (MoE) architectures are increasingly adopted in large language models (LLMs) for their scalability and efficiency. However, their modular structure introduces a unique vulnerability: adversaries can attempt to compress or repurpose models by pruning experts and cheaply fine-tuning the remainder, effectively bypassing licensing and security constraints. In this paper, we systematically study the prunability of MoE-LLMs under task-specific usage. We first develop an expert attribution framework that identifies the subset of experts most responsible for a given task, then evaluate the performance trade-offs of pruning and re-aligning these experts using active learning-driven fine-tuning. Our findings reveal a critical knowledge loss--recovery trade-off: while certain experts can be isolated to retain task accuracy, significant degradation occurs without targeted re-alignment. Based on this analysis, we propose defense strategies that aim to make MoE models harder to compress and fine-tune without authorization, including entangled expert training and selective fine-tuning protocols that resist unauthorized adaptation. By positioning expert pruning as both a threat vector and a defense target, this work highlights the dual-use nature of MoE modularity and provides the first systematic evaluation framework for secure specialization of MoE-LLMs.