FedP$^2$EFT: Federated Learning to Personalize PEFT for Multilingual LLMs

TL;DR

FedP2EFT tackles language-specific personalization in federated multilingual LLMs by learning per-client PEFT configurations through a federated PS generator and Bayesian sparse rank selection. It introduces BT-LoRA, which uses per-rank latent scales $\lambda$ and penalties on adapter weights to determine which LoRA ranks to activate, and trains a PSG in a federated fashion to predict client-specific $\bm{\lambda}$ given metadata. The framework demonstrates improvements over standard FL and personalized FL baselines on XNLI, MasakhaNEWS, and Fed-Aya across seen/unseen clients and multiple LoRA budgets, aided by language-agnostic rank structures and interpretable cross-language patterns. FedP2EFT remains compatible with various FL strategies and baselines, offering a practical, scalable pathway to personalized PEFT in multilingual settings, with manageable training and inference costs and insights into how languages influence adapter structures.

Abstract

Federated learning (FL) has enabled the training of multilingual large language models (LLMs) on diverse and decentralized multilingual data, especially on low-resource languages. To improve client-specific performance, personalization via the use of parameter-efficient fine-tuning (PEFT) modules such as LoRA is common. This involves a personalization strategy (PS), such as the design of the PEFT adapter structures (e.g., in which layers to add LoRAs and what ranks) and choice of hyperparameters (e.g., learning rates) for fine-tuning. Instead of manual PS configuration, we propose FedP$^2$EFT, a federated learning-to-personalize method for multilingual LLMs in cross-device FL settings. Unlike most existing PEFT structure selection methods, which are prone to overfitting low-data regimes, FedP$^2$EFT collaboratively learns the optimal personalized PEFT structure for each client via Bayesian sparse rank selection. Evaluations on both simulated and real-world multilingual FL benchmarks demonstrate that FedP$^2$EFT largely outperforms existing personalized fine-tuning methods, while complementing other existing FL methods.

Figures (16)

• Figure 1: (a) We train our personalization strategy generator (PSG) using standard FL approaches. (b) FedP2EFT's inference stage on a single client. Given the base model and the client's train dataset, features are extracted and passed into our PSG to generate a PS, $\bm{\lambda}$, for the client's budget. $\bm{\lambda}$ is then used to initialize all LoRA modules before the base model is personalized. The resulting personalized model is then used to evaluate on the client's test samples.
• Figure 2: FedP2EFT's federated training of PSG for each federated round. Details in Section. \ref{['sec:main_method']}.
• Figure 3: No. of clients by predominant language in our Fed-Aya setup
• Figure 4: Cross-lingual $\bm{\lambda}$ distance in our XNLI setup (See Appendix Fig. \ref{['fig:masakha_out']} for our MasakhaNEWS setup). Each block shows the log-scale normalized average Euclidean distances between all pairs of clients' $\bm{\lambda}$ in their respective languages. The smaller the distance, the more similar $\bm{\lambda}$ is.
• Figure 5: Language agnostic rank structure of mBERT in our XNLI setup where the base model is trained with Standard FL full-finetuning ($r=16$).