Stabilizing Decentralized Federated Fine-Tuning via Topology-Aware Alternating LoRA

TL;DR

This work tackles stability challenges in decentralized federated fine-tuning when using LoRA, whose bilinear factorization introduces cross-term interference under asynchronous peer-to-peer mixing. It introduces TAD-LoRA, a topology-aware framework that jointly mixes both LoRA blocks and uses interval-based directional switching to stabilize alternating LoRA in decentralized settings, supported by convergence analysis for non-convex objectives. The theoretical results reveal a topology-dependent trade-off: increasing the switching interval $T$ reduces topology-induced cross-term error at the cost of representation bias, with an optimal $T^\star(\rho) = \Theta\left(1/\sqrt{1-\rho}\right)$ tied to the network spectral gap. Empirically, TAD-LoRA achieves robust performance across diverse communication regimes, surpassing baselines as communication becomes sparser and delivering notable gains on MNLI, thereby enabling more reliable privacy-preserving decentralized fine-tuning.

Abstract

Decentralized federated learning (DFL), a serverless variant of federated learning, poses unique challenges for parameter-efficient fine-tuning due to the factorized structure of low-rank adaptation (LoRA). Unlike linear parameters, decentralized aggregation of LoRA updates introduces topology-dependent cross terms that can destabilize training under dynamic communication graphs. We propose \texttt{TAD-LoRA}, a Topology-Aware Decentralized Low-Rank Adaptation framework that coordinates the updates and mixing of LoRA factors to control inter-client misalignment. We theoretically prove the convergence of \texttt{TAD-LoRA} under non-convex objectives, explicitly characterizing the trade-off between topology-induced cross-term error and block-coordinate representation bias governed by the switching interval of alternative training. Experiments under various communication conditions validate our analysis, showing that \texttt{TAD-LoRA} achieves robust performance across different communication scenarios, remaining competitive in strongly connected topologies and delivering clear gains under moderately and weakly connected topologies, with particularly strong results on the MNLI dataset.

Figures (5)

• Figure 1: Overall illustration of TAD-LoRA under decentralized federated learning (DFL). Clients communicate directly with a dynamically selected subset of peers in a peer-to-peer manner, as determined by the underlying communication topology. At each alternating phase (lasting $T$ rounds), A-blocks or B-blocks is actively updated and coordinated across clients, while the other remains frozen during that phase and is aligned implicitly through topology-aware switching over time.
• Figure 2: Average test accuracy across datasets under different communication probabilities $p$. LoRA denotes the vanilla decentralized LoRA baseline with FedAvgmcmahan2017communication; FFA-LoRA fixes one LoRA factor during training; RoLoRA alternates LoRA factors in a naive round-robin manner; TAD-LoRA is our topology-aware decentralized alternating LoRA. As communication becomes weaker (smaller $p$), TAD-LoRA consistently outperforms all baselines, while remaining competitive under strong communication.
• Figure 3: Dataset-wise optimal switching interval $\hat{T}^*(p)$ (thin dashed lines) and the Median trend across datasets (thick line). The median aggregation mitigates the high variance introduced by argmax-based selection over a discrete candidate set. In the reliably convergent regime $p \ge 0.02$, the Median trend shifts toward larger $T$ as communication becomes weaker, consistent with the monotonicity predicted by our theory.
• Figure 4: Accuracy gain of TAD-LoRA over the LoRA baseline on MNLI under different communication probabilities $p$ and switching intervals $T$. Positive values indicate performance improvement over LoRA. Under weak communication (small $p$), performance gains concentrate on moderate to larger $T$ and span a wider range of effective switching intervals.
• Figure 5: Illustrative example of the non-monotonic dependence on the switching interval $T$ on MNLI under a weak communication regime ($p=0.02$). The y-axis reports the accuracy gain of TAD-LoRA over the LoRA baseline. Due to discrete scheduling and training noise, the U-shaped behavior predicted by theory manifests as a noisy and flattened trend rather than a sharp optimum.

Theorems & Definitions (12)

• Theorem 5.3: Convergence Rate and Trade-off
• Lemma A.4: Per-block consensus
• proof
• Proposition A.5: Cycle-averaged cross-term decay
• Lemma A.6: One-step descent of the averaged model
• proof
• Theorem A.7: Stationarity under alternating LoRA
• Theorem A.8: Function-value gap
• Corollary A.9: Optimal switching interval
• Lemma A.10: Edge-activation gossip implies a spectral gap scaling