Low Rank Comes with Low Security: Gradient Assembly Poisoning Attacks against Distributed LoRA-based LLM Systems

TL;DR

The paper identifies a structural vulnerability in FedIT-style federated LoRA fine-tuning where independent validation of low-rank adapters $(A,B)$ fails to constrain their product $\Delta W=AB$, enabling Gradient Assembly Poisoning (GAP). It formalizes four vulnerability surfaces and presents a two-stage attack that offline constructs target adapters and online injects them under stealth constraints, propagating malicious effects through the composed update across rounds and layers. Empirical evaluation shows GAP induces targeted degradation in multiple LLMs (e.g., BLEU drops up to 14.5%, substantial increases in factual/grammatical errors) while preserving fluency and long-form output length, and it evades standard detectors. The work highlights a new class of stealthy, persistent threats in distributed LoRA fine-tuning and suggests defense strategies such as composition-aware verification and adaptive layer-wise validation to mitigate such risks, with implications for the security of parameter-efficient distributed learning systems.

Abstract

Low-Rank Adaptation (LoRA) has become a popular solution for fine-tuning large language models (LLMs) in federated settings, dramatically reducing update costs by introducing trainable low-rank matrices. However, when integrated with frameworks like FedIT, LoRA introduces a critical vulnerability: clients submit $A$ and $B$ matrices separately, while only their product $AB$ determines the model update, yet this composite is never directly verified. We propose Gradient Assembly Poisoning (GAP), a novel attack that exploits this blind spot by crafting individually benign $A$ and $B$ matrices whose product yields malicious updates. GAP operates without access to training data or inter-client coordination and remains undetected by standard anomaly detectors. We identify four systemic vulnerabilities in LoRA-based federated systems and validate GAP across LLaMA, ChatGLM, and GPT-2. GAP consistently induces degraded or biased outputs while preserving surface fluency, reducing BLEU by up to 14.5\%, increasing factual and grammatical errors by over 800\%, and maintaining 92.6\% long-form response length. These results reveal a new class of stealthy, persistent threats in distributed LoRA fine-tuning.

Figures (8)

• Figure 1: Security blind spot in FedIT zhang2024towards: Individual matrix verification (green checks) misses composite malicious effects in $\Delta W = A_iB_i$
• Figure 2: Gradient Assembly Poisoning Attack workflow with the following steps: ❶ Adversarial fine-tuning; ❷ Generating target parameters; ❸ Constraint calculation; ❹ Optimization; ❺ Submiting parameters; ❻ Global model Update; ❼ Malicious output.
• Figure 3: Loss Value under various amount of Malicious Attackers
• Figure 4: Evolution of poisoning rate across training rounds for ChatGLM-6B and LLaMA-13B model.
• Figure 5: Stacked bar chart comparing topic-wise poisoning rates between DataPoison and GAP methods.