Aggressive Compression Enables LLM Weight Theft

TL;DR

The paper analyzes weight exfiltration risk for large language models by showing that extreme compression of model weights, if decompression constraints are relaxed, can dramatically accelerate theft. It introduces a simple quantitative model using $T = \frac{M c}{E N s}$ and $P_{\text{success}} = (1 - p)^T$ to connect compression, usage profiles, and detection to exfiltration feasibility, and demonstrates that $16\times$ to $100\times$ compression is achievable with minimal degradation. A novel compression method, VQ-then-train, achieves substantially higher compression than prior schemes and improves exfiltration feasibility under realistic attacker assumptions. The study also evaluates defenses, finding that forensic watermarking offers effective, low-cost attribution while other defenses (compressibility resistance and moving-target strategies) have mixed effectiveness due to adaptive attacks such as canonicalisation. Overall, the work highlights a critical security risk for weight theft and suggests actionable defense directions to mitigate exfiltration in deployed LLM systems.

Abstract

As frontier AIs become more powerful and costly to develop, adversaries have increasing incentives to steal model weights by mounting exfiltration attacks. In this work, we consider exfiltration attacks where an adversary attempts to sneak model weights out of a datacenter over a network. While exfiltration attacks are multi-step cyber attacks, we demonstrate that a single factor, the compressibility of model weights, significantly heightens exfiltration risk for large language models (LLMs). We tailor compression specifically for exfiltration by relaxing decompression constraints and demonstrate that attackers could achieve 16x to 100x compression with minimal trade-offs, reducing the time it would take for an attacker to illicitly transmit model weights from the defender's server from months to days. Finally, we study defenses designed to reduce exfiltration risk in three distinct ways: making models harder to compress, making them harder to 'find,' and tracking provenance for post-attack analysis using forensic watermarks. While all defenses are promising, the forensic watermark defense is both effective and cheap, and therefore is a particularly attractive lever for mitigating weight-exfiltration risk.

Figures (12)

• Figure 1: Prior compression methods are optimized for efficient inference, with minimal decompression overhead. By relaxing these constraints, we show that much smaller model weights can be achieved. This is highly relevant for weight exfiltration, as it reduces both exfiltration time and detection risk. After being stolen from the server, the model can be decompressed and fine-tuned to recover performance at far lower cost than training from scratch.
• Figure 2: Model compression enables more successful weight exfiltration attacks. We compute the rate of weight exfiltration (left) and the probability of detection (right) for full models and models compressed using our method designed for the weight exfiltration setting. Probability of detection is computed from days to exfiltrate. For more details, see \ref{['sec:quantitative_model']}.
• Figure 3: The MSE between the original and compressed weights vs the bits-per-parameter (BPP), along with the Pareto frontier for MSE vs BPP, for each Pythia model. We observe that in weight space, larger models are easier to compress.
• Figure 4: We study three defenses against weight exfiltration: Left: fine-tuning weights to make them harder to compress-to-steal; Middle: adding watermarks to weights to attribute the data and time of exfiltration; Right: a moving target defense that attempts to use model symmetries to make the correct weights more difficult to piece together.
• Figure 5: Full MSE vs BPP (rate-distortion) plot of a standard Qwen2.5-1.5B vs a Qwen2.5-1.5B model layers finetuned to be harder to compress.