Collaborative and Efficient Personalization with Mixtures of Adaptors
Abdulla Jasem Almansoori, Samuel Horváth, Martin Takáč
TL;DR
This paper introduces FLoRAL, a parameter-efficient federated personalization framework that represents personalized models as a shared base with a small set of low-rank adaptors and a client-specific mixture router. It provides a convergence analysis that accounts for aggregation mismatch and demonstrates that weight sharing can yield variance reduction, enabling robust generalization in data-scarce and heterogeneous settings. Empirically, FLoRAL outperforms baselines such as FedAvg, Local Adaptor, and Ensemble across synthetic, MNIST/CIFAR-10, and CIFAR-100 clustered tasks, while using substantially fewer additional parameters than full-model ensembles. The approach offers practical benefits for cross-device FL by enabling scalable, collaborative personalization with provable guarantees and broad applicability to convolutional and linear layers, including ConvLoRA variants.
Abstract
Heterogenous data is prevalent in real-world federated learning. We propose a parameter-efficient framework, Federated Low-Rank Adaptive Learning (FLoRAL), that allows clients to personalize in groups by mixing between low-rank adaptors, where the mixtures are client-specific. FLoRAL is a model parameterization that casts personalized federated learning as a multi-task learning problem, with weight sharing as an implicit regularizer. It is memory-efficient, as the personalized parameters (i.e., base model + adaptors) are all federated. Our results show that FLoRAL can generalize better than a mixture of full models when data are scarce. It can also consistently personalize better than models with a locally tuned adaptor per client. This demonstrates the benefits of "federated personalization" and its robustness against overfitting. We derive the convergence rates and show theoretically that FLoRAL can lead to better variance reduction of the base model's gradients.