Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FLoCoRA: Federated learning compression with low-rank adaptation

Lucas Grativol Ribeiro, Mathieu Leonardon, Guillaume Muller, Virginie Fresse, Matthieu Arzel

TL;DR

This paper tackles the high communication and memory costs of Federated Learning by applying Low-Rank Adaptation (LoRA) through learnable adapters, enabling small vision models to be trained from scratch without updating the base parameters. The proposed FLoCoRA framework freezes the original network and exchanges low-rank adapters, achieving up to 4.8x communication reduction with <1% accuracy loss on CIFAR-10 (ResNet-8) and, when paired with affine quantization, up to 18.6x further reduction for ResNet-18. The approach remains aggregation-agnostic and demonstrates compatibility with additional compression, offering strong baseline performance for message-size reduction while also cutting training memory requirements. These results suggest promising directions for deploying FL on edge devices and motivate further exploration of rank and quantization strategies in heterogeneous client settings.

Abstract

Low-Rank Adaptation (LoRA) methods have gained popularity in efficient parameter fine-tuning of models containing hundreds of billions of parameters. In this work, instead, we demonstrate the application of LoRA methods to train small-vision models in Federated Learning (FL) from scratch. We first propose an aggregation-agnostic method to integrate LoRA within FL, named FLoCoRA, showing that the method is capable of reducing communication costs by 4.8 times, while having less than 1% accuracy degradation, for a CIFAR-10 classification task with a ResNet-8. Next, we show that the same method can be extended with an affine quantization scheme, dividing the communication cost by 18.6 times, while comparing it with the standard method, with still less than 1% of accuracy loss, tested with on a ResNet-18 model. Our formulation represents a strong baseline for message size reduction, even when compared to conventional model compression works, while also reducing the training memory requirements due to the low-rank adaptation.

FLoCoRA: Federated learning compression with low-rank adaptation

TL;DR

This paper tackles the high communication and memory costs of Federated Learning by applying Low-Rank Adaptation (LoRA) through learnable adapters, enabling small vision models to be trained from scratch without updating the base parameters. The proposed FLoCoRA framework freezes the original network and exchanges low-rank adapters, achieving up to 4.8x communication reduction with <1% accuracy loss on CIFAR-10 (ResNet-8) and, when paired with affine quantization, up to 18.6x further reduction for ResNet-18. The approach remains aggregation-agnostic and demonstrates compatibility with additional compression, offering strong baseline performance for message-size reduction while also cutting training memory requirements. These results suggest promising directions for deploying FL on edge devices and motivate further exploration of rank and quantization strategies in heterogeneous client settings.

Abstract

Low-Rank Adaptation (LoRA) methods have gained popularity in efficient parameter fine-tuning of models containing hundreds of billions of parameters. In this work, instead, we demonstrate the application of LoRA methods to train small-vision models in Federated Learning (FL) from scratch. We first propose an aggregation-agnostic method to integrate LoRA within FL, named FLoCoRA, showing that the method is capable of reducing communication costs by 4.8 times, while having less than 1% accuracy degradation, for a CIFAR-10 classification task with a ResNet-8. Next, we show that the same method can be extended with an affine quantization scheme, dividing the communication cost by 18.6 times, while comparing it with the standard method, with still less than 1% of accuracy loss, tested with on a ResNet-18 model. Our formulation represents a strong baseline for message size reduction, even when compared to conventional model compression works, while also reducing the training memory requirements due to the low-rank adaptation.
Paper Structure (8 sections, 2 equations, 3 figures, 4 tables)

This paper contains 8 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Federated Learning
  4. Model Compression
  5. Low-Rank Adaptation
  6. Our method - FLoCoRA
  7. Experiments and Discussion
  8. Conclusion

Figures (3)

  • Figure 1: FLoCoRA training loop.
  • Figure 2: The relationship between the $r$ hyperparameter in FLoCoRA and the scaling factor $\alpha$, in $\frac{\alpha}{r}$. Two scenarios are evaluated, $\alpha=2r$ and $\alpha=16r$, against FedAvg.
  • Figure 3: Convergence behavior between FedAvg, FLoCoRA with rank of 32 and its quantized versions of 2/4/8-bits.