Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Promoting Data and Model Privacy in Federated Learning through Quantized LoRA

JianHao Zhu, Changze Lv, Xiaohua Wang, Muling Wu, Wenhao Liu, Tianlong Li, Zixuan Ling, Cenyuan Zhang, Xiaoqing Zheng, Xuanjing Huang

TL;DR

FedLPP tackles simultaneous data privacy and model IP protection in federated learning for large language models by distributing a quantized proxy of LoRA adapters to clients, enabling gradient updates without exposing a strong global model. It combines quantization with LoRA-based PEFT to reduce communication while preserving training effectiveness. The empirical results across multiple datasets and model sizes show FedLPP outperforms the baseline FedSP while maintaining privacy and achieving comparable performance to non-private baselines in many settings. This approach offers a practical, resource-efficient path toward privacy-preserving FL for commercial LLM deployments.

Abstract

Conventional federated learning primarily aims to secure the privacy of data distributed across multiple edge devices, with the global model dispatched to edge devices for parameter updates during the learning process. However, the development of large language models (LLMs) requires substantial data and computational resources, rendering them valuable intellectual properties for their developers and owners. To establish a mechanism that protects both data and model privacy in a federated learning context, we introduce a method that just needs to distribute a quantized version of the model's parameters during training. This method enables accurate gradient estimations for parameter updates while preventing clients from accessing a model whose performance is comparable to the centrally hosted one. Moreover, we combine this quantization strategy with LoRA, a popular and parameter-efficient fine-tuning method, to significantly reduce communication costs in federated learning. The proposed framework, named \textsc{FedLPP}, successfully ensures both data and model privacy in the federated learning context. Additionally, the learned central model exhibits good generalization and can be trained in a resource-efficient manner.

Promoting Data and Model Privacy in Federated Learning through Quantized LoRA

TL;DR

FedLPP tackles simultaneous data privacy and model IP protection in federated learning for large language models by distributing a quantized proxy of LoRA adapters to clients, enabling gradient updates without exposing a strong global model. It combines quantization with LoRA-based PEFT to reduce communication while preserving training effectiveness. The empirical results across multiple datasets and model sizes show FedLPP outperforms the baseline FedSP while maintaining privacy and achieving comparable performance to non-private baselines in many settings. This approach offers a practical, resource-efficient path toward privacy-preserving FL for commercial LLM deployments.

Abstract

Conventional federated learning primarily aims to secure the privacy of data distributed across multiple edge devices, with the global model dispatched to edge devices for parameter updates during the learning process. However, the development of large language models (LLMs) requires substantial data and computational resources, rendering them valuable intellectual properties for their developers and owners. To establish a mechanism that protects both data and model privacy in a federated learning context, we introduce a method that just needs to distribute a quantized version of the model's parameters during training. This method enables accurate gradient estimations for parameter updates while preventing clients from accessing a model whose performance is comparable to the centrally hosted one. Moreover, we combine this quantization strategy with LoRA, a popular and parameter-efficient fine-tuning method, to significantly reduce communication costs in federated learning. The proposed framework, named \textsc{FedLPP}, successfully ensures both data and model privacy in the federated learning context. Additionally, the learned central model exhibits good generalization and can be trained in a resource-efficient manner.
Paper Structure (20 sections, 4 equations, 2 figures, 5 tables)

This paper contains 20 sections, 4 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Learning
  4. LLMs-Privacy-Protection in Federated Learning
  5. Quantization for Federated Learning
  6. Parameter-Efficient Fine-Tuning for Federated Learning
  7. Methods
  8. Preliminary
  9. Computing Proxy LoRA Matrices by Quantization
  10. Client-Side Local Fine-tuning
  11. Experiments
  12. Models and Datasets
  13. Baselines
  14. Implementation Details
  15. Performance Comparison
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Visualization of the proposed method. To prevent clients (or edge devices) from fully accessing to a global model, the central server broadcasts only the quantized version of the global LoRA's parameters at each communication round. The two bar charts visually demonstrate the quantization process, wherein eight distinct parameter values are mapped into four categories (highlighted in red, green, red, purple and blue).
  • Figure 2: Performance comparison of various methods on four different datasets. As shown in the radar chart, On all four datasets, FedLPP has achieved the goal of protecting the model and demonstrated significant performance improvements compared to the baseline FedSP.