Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedRand: Enhancing Privacy in Federated Learning with Randomized LoRA Subparameter Updates

Sangwoo Park, Seanie Lee, Byungjoo Kim, Sung Ju Hwang

TL;DR

The paper tackles privacy risks in federated fine-tuning of vision-language models by exposing local parameters during aggregation. It introduces FedRand, a privacy-enhanced FL framework where each client randomly selects a subset of LoRA subparameters to share and keeps the rest private, thereby mitigating membership inference attacks while maintaining competitive task accuracy. Empirical results on ScienceQA, MSCOCO, and NoCaps show FedRand achieves comparable performance to FedAvg with substantially improved robustness to MIAs and reduced communication costs. Ablation studies validate the chosen sampling rate and normalization, and the work highlights practical implications for privacy-preserving FL in multimodal settings and potential directions like sub-layer randomization and parameter quantization.

Abstract

Federated Learning (FL) is a widely used framework for training models in a decentralized manner, ensuring that the central server does not have direct access to data from local clients. However, this approach may still fail to fully preserve data privacy, as models from local clients are exposed to the central server during the aggregation process. This issue becomes even more critical when training vision-language models (VLMs) with FL, as VLMs can easily memorize training data instances, making them vulnerable to membership inference attacks (MIAs). To address this challenge, we propose the FedRand framework, which avoids disclosing the full set of client parameters. In this framework, each client randomly selects subparameters of Low-Rank Adaptation (LoRA) from the server and keeps the remaining counterparts of the LoRA weights as private parameters. After training both parameters on the client's private dataset, only the non-private client parameters are sent back to the server for aggregation. This approach mitigates the risk of exposing client-side VLM parameters, thereby enhancing data privacy. We empirically validate that FedRand improves robustness against MIAs compared to relevant baselines while achieving accuracy comparable to methods that communicate full LoRA parameters across several benchmark datasets.

FedRand: Enhancing Privacy in Federated Learning with Randomized LoRA Subparameter Updates

TL;DR

The paper tackles privacy risks in federated fine-tuning of vision-language models by exposing local parameters during aggregation. It introduces FedRand, a privacy-enhanced FL framework where each client randomly selects a subset of LoRA subparameters to share and keeps the rest private, thereby mitigating membership inference attacks while maintaining competitive task accuracy. Empirical results on ScienceQA, MSCOCO, and NoCaps show FedRand achieves comparable performance to FedAvg with substantially improved robustness to MIAs and reduced communication costs. Ablation studies validate the chosen sampling rate and normalization, and the work highlights practical implications for privacy-preserving FL in multimodal settings and potential directions like sub-layer randomization and parameter quantization.

Abstract

Federated Learning (FL) is a widely used framework for training models in a decentralized manner, ensuring that the central server does not have direct access to data from local clients. However, this approach may still fail to fully preserve data privacy, as models from local clients are exposed to the central server during the aggregation process. This issue becomes even more critical when training vision-language models (VLMs) with FL, as VLMs can easily memorize training data instances, making them vulnerable to membership inference attacks (MIAs). To address this challenge, we propose the FedRand framework, which avoids disclosing the full set of client parameters. In this framework, each client randomly selects subparameters of Low-Rank Adaptation (LoRA) from the server and keeps the remaining counterparts of the LoRA weights as private parameters. After training both parameters on the client's private dataset, only the non-private client parameters are sent back to the server for aggregation. This approach mitigates the risk of exposing client-side VLM parameters, thereby enhancing data privacy. We empirically validate that FedRand improves robustness against MIAs compared to relevant baselines while achieving accuracy comparable to methods that communicate full LoRA parameters across several benchmark datasets.

Paper Structure

This paper contains 25 sections, 8 equations, 6 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated learning.
  4. Membership inference attack.
  5. Method
  6. Preliminaries
  7. Privacy Enhanced FL: FedRand
  8. Experiments
  9. Setup
  10. Dataset.
  11. Evaluation metrics.
  12. Implementation details.
  13. Baselines.
  14. Experimental Results
  15. Main results.
  16. ...and 10 more sections

Figures (6)

  • Figure 1: (a). At each round $r$, each local client selects a LoRA weight either $A_r$ or $B_r$ for initialization from the server and initializes the other counterparts of LoRA weights using the previous round's client model parameters as private parameters. (b). After updating both parameters, only the non-private parameters are sent back to the server and aggregated to update the LoRA weights of the central server.
  • Figure 2: Trade-off between task performance (CIDEr) and vulnerability to membership inference attacks (AUROC of MIA) on MSCOCO dataset.
  • Figure 3: The ratio of number of communicated LoRA parameters, compared to FedAvg per round under LoRA configuration.
  • Figure 4: An example of token-wise Rényi entropy measurement for member (MSCOCO) and non-member (NoCaps) data. The higher the entropy is, the more robust to MIA.
  • Figure : FedRand
  • ...and 1 more figures