Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Label Privacy in Split Learning for Large Models with Parameter-Efficient Training

Philip Zmushko, Marat Mansurov, Ruslan Svirschevski, Denis Kuznedelev, Max Ryabinin, Aleksandr Beznosikov

TL;DR

The paper tackles label privacy in API-based fine-tuning of large models by modeling a two-party split-learning setting with parameter-efficient fine-tuning. It analyzes the privacy risks of LoRA and introduces P$^3$EFT, a protocol that combines privacy-preserving backpropagation with multi-adapter mixing and an adversarial regularizer to obfuscate per-adapter signals while preserving task performance. Through extensive experiments on DeBERTa-XXL, Flan-T5-Large, and LLaMA-2 7B across SST-2, MNLI, and QNLI, the approach achieves competitive accuracy relative to non-private baselines and demonstrates reduced label leakage compared to existing privacy baselines like PSLF and DC. The work provides a practical pathway for private fine-tuning over APIs, highlighting implications for vertical federated learning and real-world deployment of PEFT in privacy-sensitive settings.

Abstract

As deep learning models become larger and more expensive, many practitioners turn to fine-tuning APIs. These web services allow fine-tuning a model between two parties: the client that provides the data, and the server that hosts the model. While convenient, these APIs raise a new concern: the data of the client is at risk of privacy breach during the training procedure. This challenge presents an important practical case of vertical federated learning, where the two parties perform parameter-efficient fine-tuning (PEFT) of a large model. In this study, we systematically search for a way to fine-tune models over an API while keeping the labels private. We analyze the privacy of LoRA, a popular approach for parameter-efficient fine-tuning when training over an API. Using this analysis, we propose P$^3$EFT, a multi-party split learning algorithm that takes advantage of existing PEFT properties to maintain privacy at a lower performance overhead. To validate our algorithm, we fine-tune DeBERTa-v2-XXLarge, Flan-T5 Large and LLaMA-2 7B using LoRA adapters on a range of NLP tasks. We find that P$^3$EFT is competitive with existing privacy-preserving methods in multi-party and two-party setups while having higher accuracy.

Label Privacy in Split Learning for Large Models with Parameter-Efficient Training

TL;DR

The paper tackles label privacy in API-based fine-tuning of large models by modeling a two-party split-learning setting with parameter-efficient fine-tuning. It analyzes the privacy risks of LoRA and introduces PEFT, a protocol that combines privacy-preserving backpropagation with multi-adapter mixing and an adversarial regularizer to obfuscate per-adapter signals while preserving task performance. Through extensive experiments on DeBERTa-XXL, Flan-T5-Large, and LLaMA-2 7B across SST-2, MNLI, and QNLI, the approach achieves competitive accuracy relative to non-private baselines and demonstrates reduced label leakage compared to existing privacy baselines like PSLF and DC. The work provides a practical pathway for private fine-tuning over APIs, highlighting implications for vertical federated learning and real-world deployment of PEFT in privacy-sensitive settings.

Abstract

As deep learning models become larger and more expensive, many practitioners turn to fine-tuning APIs. These web services allow fine-tuning a model between two parties: the client that provides the data, and the server that hosts the model. While convenient, these APIs raise a new concern: the data of the client is at risk of privacy breach during the training procedure. This challenge presents an important practical case of vertical federated learning, where the two parties perform parameter-efficient fine-tuning (PEFT) of a large model. In this study, we systematically search for a way to fine-tune models over an API while keeping the labels private. We analyze the privacy of LoRA, a popular approach for parameter-efficient fine-tuning when training over an API. Using this analysis, we propose PEFT, a multi-party split learning algorithm that takes advantage of existing PEFT properties to maintain privacy at a lower performance overhead. To validate our algorithm, we fine-tune DeBERTa-v2-XXLarge, Flan-T5 Large and LLaMA-2 7B using LoRA adapters on a range of NLP tasks. We find that PEFT is competitive with existing privacy-preserving methods in multi-party and two-party setups while having higher accuracy.

Paper Structure

This paper contains 17 sections, 8 equations, 3 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Federated learning and split learning
  4. Parameter-efficient finetuning
  5. Privacy-preserving parameter-efficient fine-tuning
  6. Setup
  7. Label Leakage of Standard Split Learning
  8. Privacy-preserving backpropagation
  9. Full fine-tuning
  10. Experiments
  11. Privacy of gradients
  12. Main fine-tuning experiments
  13. Conclusion and Discussion
  14. Acknowledgments
  15. Hyperparameters search
  16. ...and 2 more sections

Figures (3)

  • Figure 1: An intuitive illustration of the proposed fine-tuning protocol.
  • Figure 2: A visualization of top-2 principal components of gradients (top) and activations (bottom) from different fine-tuning steps (left to right). Color indicates the training labels (binary).
  • Figure 3: Gradients of cross-entropy w.r.t. LoRA parameters for DeBERTa-v2-XXLarge. The top row corresponds to normal backpropagation and the bottom row uses privacy-preserving backprop.