Communication-Efficient Wireless Federated Fine-Tuning for Large-Scale AI Models

TL;DR

This work tackles the problem of efficiently fine-tuning large transformers in wireless federated learning by proposing a rank-informed, parameter-efficient approach. It introduces Sparsified Orthogonal Fine-Tuning (SOFT), which enforces approximate orthogonality on LoRA factors $\theta_B \in \mathbb{R}^{d\times r}$ and $\theta_A \in \mathbb{R}^{r\times \ell}$ to enable low-cost importance Estimation without heavy matrix multiplications or SVD, and a Two-Stage Federated Algorithm (TSFA) that offline-determines the LoRA rank $r$ and online-optimizes sparsification $O^t$ and bandwidth $b_k^t$ via Lyapunov optimization. Theoretical contributions include a convergence analysis that explicitly incorporates LoRA rank and a covariance term, plus an online control scheme that preserves long-term latency constraints. Empirical results on CIFAR-100 with ViT-Base show the proposed framework achieves accuracy comparable to ideal centralized training while dramatically reducing communication overhead, enabling scalable deployment of large-scale AI models in wireless FL. Key ideas combine LoRA parameterization, covariance-aware convergence, orthogonal sparsification, and Lyapunov-driven online resource control to unlock practical wireless FL for large models.

Abstract

Transformer-based large language models (LLMs) have achieved remarkable success across various tasks. Yet, fine-tuning such massive models in federated learning (FL) settings poses significant challenges due to resource constraints and communication overhead. Low-Rank Adaptation (LoRA) addresses these issues by training compact, low-rank matrices instead of fully fine-tuning large models. This paper introduces a wireless federated LoRA fine-tuning framework that optimizes both learning performance and communication efficiency. We provide a novel convergence analysis, revealing how LoRA rank and covariance effects influence FL training dynamics. Leveraging these insights, we propose Sparsified Orthogonal Fine-Tuning (\textbf{SOFT}), an adaptive sparsification method that streamlines parameter updates without expensive matrix multiplications and singular value decomposition (SVD) operations. Additionally, we present a Two Stage Federated Algorithm (\textbf{TSFA}) algorithm that pre-determines key parameters offline and dynamically adjusts bandwidth and sparsification online, ensuring efficient training under latency constraints. Experiments on benchmark datasets show that our approach achieves accuracy comparable to ideal scenario models while significantly reducing communication overhead. Our framework thus enables scalable, resource-efficient deployment of large models in real-world wireless FL scenarios.

Figures (5)

• Figure 1: System model of Federated LoRA Fine-Tuning.
• Figure 2: Visualization of $\theta_{A,k}\theta_{A,k}^\textsf{T}$ of Vit-Base blocks.6.mlp.fc1 layer: without (a) and with (b) the proposed loss function.
• Figure 3: Test accuracy of CIFAR-100 classification. (a) and (b) show the performance using LoRA with ranks $r=4,r=8$, respectively, under various sparsification methods, while panel (c) compares the performance of OSFA and TSFA.
• Figure 4: Frobenius norm of covariance under different number of shards.
• Figure 5: Test accuracy of CIFAR-100 under different number of shards.

Theorems & Definitions (8)

• Lemma 1
• Corollary 1
• Remark 1
• Remark 2
• Theorem 1
• Remark 3
• Lemma 2
• Lemma 3