Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SDFLoRA: Selective Dual-Module LoRA for Federated Fine-tuning with Heterogeneous Clients

Zhikang Shen, Jianrong Lu, Haiyuan Wan, Jianhai Chen

TL;DR

Selective Dual-module Federated LoRA (SDFLoRA) is proposed, which decomposes each client adapter into a global module that captures transferable knowledge and a local module that preserves client-specific adaptations.

Abstract

Federated learning (FL) for large language models (LLMs) has attracted increasing attention as a way to enable privacy-preserving adaptation over distributed data. Parameter-efficient methods such as LoRA are widely adopted to reduce communication and memory costs. Despite these advances, practical FL deployments often exhibit rank heterogeneity, since different clients may use different low-rank configurations. This makes direct aggregation of LoRA updates biased and unstable. Existing solutions typically enforce unified ranks or align heterogeneous updates into a shared subspace, which over-constrains client-specific semantics, limits personalization, and provides weak protection of local client information under differential privacy noise. To address this issue, we propose Selective Dual-module Federated LoRA (SDFLoRA), which decomposes each client adapter into a global module that captures transferable knowledge and a local module that preserves client-specific adaptations. The global module is selectively aligned and aggregated across clients, while local modules remain private. This design enables robust learning under rank heterogeneity and supports privacy-aware optimization by injecting differential privacy noise exclusively into the global module. Experiments on GLUE benchmarks demonstrate that SDFLoRA outperforms representative federated LoRA baselines and achieves a better utility-privacy trade-off.

SDFLoRA: Selective Dual-Module LoRA for Federated Fine-tuning with Heterogeneous Clients

TL;DR

Selective Dual-module Federated LoRA (SDFLoRA) is proposed, which decomposes each client adapter into a global module that captures transferable knowledge and a local module that preserves client-specific adaptations.

Abstract

Federated learning (FL) for large language models (LLMs) has attracted increasing attention as a way to enable privacy-preserving adaptation over distributed data. Parameter-efficient methods such as LoRA are widely adopted to reduce communication and memory costs. Despite these advances, practical FL deployments often exhibit rank heterogeneity, since different clients may use different low-rank configurations. This makes direct aggregation of LoRA updates biased and unstable. Existing solutions typically enforce unified ranks or align heterogeneous updates into a shared subspace, which over-constrains client-specific semantics, limits personalization, and provides weak protection of local client information under differential privacy noise. To address this issue, we propose Selective Dual-module Federated LoRA (SDFLoRA), which decomposes each client adapter into a global module that captures transferable knowledge and a local module that preserves client-specific adaptations. The global module is selectively aligned and aggregated across clients, while local modules remain private. This design enables robust learning under rank heterogeneity and supports privacy-aware optimization by injecting differential privacy noise exclusively into the global module. Experiments on GLUE benchmarks demonstrate that SDFLoRA outperforms representative federated LoRA baselines and achieves a better utility-privacy trade-off.
Paper Structure (38 sections, 9 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 38 sections, 9 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Optimization under Client Heterogeneity
  4. Parameter-Efficient Fine-Tuning and Federated LoRA
  5. Personalization and Privacy in Federated Learning
  6. Positioning of Our Work
  7. Method
  8. Preliminaries: Federated LoRA Fine-tuning
  9. Challenge: rank heterogeneity and semantic heterogeneity.
  10. Dual-Module LoRA Decomposition
  11. Federated roles.
  12. Optimization objective.
  13. Selective Stacking Aggregation on the Global Module
  14. Stacking construction.
  15. Rank control via re-compression.
  16. ...and 23 more sections

Figures (4)

  • Figure 1: Framework overview. Each client keeps a private (local) module and trains a public (global) adapter on a frozen LLM. Only the public module is communicated and aggregated via selective stacking; DP noise is applied to the public update.
  • Figure 2: Test accuracy under varying numbers of federated clients ($K$) for two rank settings. Increasing $K$ generally improves performance due to broader data coverage, while non-IID update discrepancy can cause transient fluctuations.
  • Figure 3: Client-level performance comparison between single-module and dual-module adapters under the same training budget. The dual-module design consistently improves accuracy across clients, validating the effectiveness of separating global and local low-rank adaptations.
  • Figure 4: Accuracy under different privacy budgets $\epsilon$. Fixed denotes applying structural constraints (selective stacking and rank control) on the global module, while Unfixed corresponds to unconstrained aggregation over all updates. The fixed design exhibits substantially better stability under strong privacy constraints.