SHE-LoRA: Selective Homomorphic Encryption for Federated Tuning with Heterogeneous LoRA

TL;DR

SHE-LoRA tackles privacy risks in federated fine-tuning of large language models by adaptively encrypting only the most sensitive LoRA parameters. It combines selective homomorphic encryption with a column-aware aggregation and SVD-based reparameterization to accommodate device heterogeneity, maintaining performance while defending against inversion attacks such as DAGER. The main contributions are a globally negotiated encryption subset, column-swapping obfuscation, and a two-stage aggregation/reparameterization pipeline that preserves LoRA structure across heterogeneous clients. The results show strong privacy protection with minimal impact on utility and substantial reductions in communication and encryption overhead, enabling practical privacy-preserving federated PEFT in cross-device settings.

Abstract

Federated fine-tuning of large language models (LLMs) is critical for improving their performance in handling domain-specific tasks. However, prior work has shown that clients' private data can actually be recovered via gradient inversion attacks. Existing privacy preservation techniques against such attacks typically entail performance degradation and high costs, making them ill-suited for clients with heterogeneous data distributions and device capabilities. In this paper, we propose SHE-LoRA, which integrates selective homomorphic encryption (HE) and low-rank adaptation (LoRA) to enable efficient and privacy-preserving federated tuning of LLMs in cross-device environment. Heterogeneous clients adaptively select partial model parameters for homomorphic encryption based on parameter sensitivity assessment, with the encryption subset obtained via negotiation. To ensure accurate model aggregation, we design a column-aware secure aggregation method and customized reparameterization techniques to align the aggregation results with the heterogeneous device capabilities of clients. Extensive experiments demonstrate that SHE-LoRA maintains performance comparable to non-private baselines, achieves strong resistance to the state-of-the-art attacks, and significantly reduces communication overhead by 94.901\% and encryption computation overhead by 99.829\%, compared to baseline. Our code is accessible at https://anonymous.4open.science/r/SHE-LoRA-8D84.

Figures (13)

• Figure 1: Expansion of encryption positions.
• Figure 2: Inflation of ciphertext size.
• Figure 3: The diagram of the SHE-LoRA.
• Figure 4: Sensitivity of model parameters.
• Figure 5: Selective encryption of columns.