Reconstructing Training Data from Adapter-based Federated Large Language Models

TL;DR

This work reveals that adapter-based Federated LLMs, despite freezing backbones and sharing only compact adapter updates, remain vulnerable to gradient inversion attacks. It introduces UTR, a two-stage attack that first infers a word bag from embedding gradients and then reconstructs coherent text under a constrained search guided by low-rank subspaces in the adapter gradients; this yields near-perfect reconstructions across multiple models and datasets, even for large batch sizes. The authors characterize the attack’s theoretical limits via a rank-based bound on recoverable tokens and demonstrate that common defenses like differential privacy can mitigate leakage only at substantial cost to model utility, while gradient pruning remains brittle. Overall, the paper demonstrates a fundamental privacy–efficiency trade-off in adapter-based FedLLMs and calls for robust privacy guarantees beyond parameter efficiency.

Abstract

Adapter-based Federated Large Language Models (FedLLMs) are widely adopted to reduce the computational, storage, and communication overhead of full-parameter fine-tuning for web-scale applications while preserving user privacy. By freezing the backbone and training only compact low-rank adapters, these methods appear to limit gradient leakage and thwart existing Gradient Inversion Attacks (GIAs). Contrary to this assumption, we show that low-rank adapters create new, exploitable leakage channels. We propose the Unordered-word-bag-based Text Reconstruction (UTR) attack, a novel GIA tailored to the unique structure of adapter-based FedLLMs. UTR overcomes three core challenges: low-dimensional gradients, frozen backbones, and combinatorially large reconstruction spaces by: (i) inferring token presence from attention patterns in frozen layers, (ii) performing sentence-level inversion within the low-rank subspace of adapter gradients, and (iii) enforcing semantic coherence through constrained greedy decoding guided by language priors. Extensive experiments across diverse models (GPT2-Large, BERT, Qwen2.5-7B) and datasets (CoLA, SST-2, Rotten Tomatoes) demonstrate that UTR achieves near-perfect reconstruction accuracy (ROUGE-1/2 > 99), even with large batch size settings where prior GIAs fail completely. Our results reveal a fundamental tension between parameter efficiency and privacy in FedLLMs, challenging the prevailing belief that lightweight adaptation inherently enhances security. Our code and data are available at https://github.com/shwksnshwowk-wq/GIA.

Figures (5)

• Figure 1: Standard architectures of adapter-based FedLLMs.
• Figure 2: Overview of the proposed UTR attack.
• Figure 3: Relationship between the number of recovered tokens in the gradient information of the embedding adapter and the number of true tokens.
• Figure 4: The performance across various $reduction_{factor}$ and $atol$ settings.
• Figure 5: The showcase of the UTR attack.

Theorems & Definitions (3)

• theorem 1
• theorem 2
• corollary 1