Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning

Xiaoyu Wang, Xiaotian Li, Zhixiang Zhou, Chen Li, Yong Liu

TL;DR

ADF-LoRA is introduced, which synchronizes the update of only one low-rank matrix per round and mixes both matrices to maintain more consistent parameter states under decentralized propagation and achieves faster and smoother convergence and delivers the highest average accuracy across tasks.

Abstract

This paper revisits alternating low-rank updates for federated fine-tuning and examines their behavior in decentralized federated learning (DFL). While alternating the LoRA matrices has been shown to stabilize aggregation in centralized FL, extending this mechanism to decentralized, peer-to-peer communication introduces new challenges due to phase-state mismatch and block-wise divergence across clients. We introduce ADF-LoRA, which synchronizes the update of only one low-rank matrix per round and mixes both matrices to maintain more consistent parameter states under decentralized propagation. This design preserves the cross-term suppression effect of alternating updates while improving stability in serverless topologies. We provide a convergence analysis under standard smoothness assumptions and evaluate ADF-LoRA on multiple GLUE tasks. Experiments show that ADF-LoRA achieves faster and smoother convergence and delivers the highest average accuracy across tasks, outperforming existing LoRA variants in decentralized FL by a consistent margin.

ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning

TL;DR

ADF-LoRA is introduced, which synchronizes the update of only one low-rank matrix per round and mixes both matrices to maintain more consistent parameter states under decentralized propagation and achieves faster and smoother convergence and delivers the highest average accuracy across tasks.

Abstract

This paper revisits alternating low-rank updates for federated fine-tuning and examines their behavior in decentralized federated learning (DFL). While alternating the LoRA matrices has been shown to stabilize aggregation in centralized FL, extending this mechanism to decentralized, peer-to-peer communication introduces new challenges due to phase-state mismatch and block-wise divergence across clients. We introduce ADF-LoRA, which synchronizes the update of only one low-rank matrix per round and mixes both matrices to maintain more consistent parameter states under decentralized propagation. This design preserves the cross-term suppression effect of alternating updates while improving stability in serverless topologies. We provide a convergence analysis under standard smoothness assumptions and evaluate ADF-LoRA on multiple GLUE tasks. Experiments show that ADF-LoRA achieves faster and smoother convergence and delivers the highest average accuracy across tasks, outperforming existing LoRA variants in decentralized FL by a consistent margin.

Paper Structure

This paper contains 33 sections, 5 theorems, 44 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Federated Learning
  5. Federated Learning
  6. LoRA in Federated Learning
  7. Motivation and Analysis
  8. Why alternating works in CFL?
  9. Why this breaks in DFL?
  10. Methodology
  11. Decentralized FL Communication Model
  12. Alternating LoRA in FL
  13. CFL: Centralized Alternating LoRA (RoLoRA).
  14. DFL: Decentralized Alternating LoRA Baseline.
  15. ADF-LoRA: Joint-Mixing Alternating LoRA
  16. ...and 18 more sections

Key Result

Theorem 4.1

After $K$ full periods ($2KT$ block updates), the averaged iterate satisfies Moreover, since the consensus error term vanishes geometrically as $t\to\infty$, and thus $\nabla\mathcal{L}(\bar{A}^t,\bar{B}^t)\to 0$.

Figures (3)

  • Figure 1: Accuracy and standard deviation across seeds for LoRA variants on four GLUE tasks under centralized FL.
  • Figure 2: Convergence curves for QNLI under DFL.
  • Figure 3: Effect of interval $T$ on convergence on QNLI.

Theorems & Definitions (10)

  • Theorem 4.1: Decentralized Convergence
  • Definition A.1: $L$-smoothness xie2019asynchronous
  • Definition A.2: $\mu$-weak convexity xie2019asynchronous
  • Lemma A.1: Consensus Error Decay Under Joint Mixing
  • Lemma A.2: Descent with Consensus Error
  • proof
  • Theorem A.1: Decentralized Convergence
  • proof
  • Theorem A.2: Convergence in Centralized FL
  • proof