Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPC

Jinglong Luo, Zhuo Zhang, Yehong Zhang, Shiyu Liu, Ye Dong, Hui Wang, Yue Yu, Xun Zhou, Zenglin Xu

TL;DR

SecP-Tuning is proposed, the first MPC-based framework designed for efficient, privacy-preserving prompt tuning of LLMs, and devises an efficient privacy-preserving Random Feature Attention (RFA), effectively mitigating the computational complexity of softmax-based self-attention and circumventing MPC-incompatible nonlinear operations.

Abstract

Large Language Models (LLMs) have revolutionized numerous fields, yet their adaptation to specialized tasks in privacy-sensitive domains such as healthcare and finance remains constrained due to the scarcity of accessible training data caused by stringent privacy requirements. Secure Multi-party Computation (MPC)-based privacy-preserving machine learning provides theoretical guarantees for the privacy of model parameters and data. However, its application to LLMs has been predominantly limited to inference, as fine-tuning introduces significant efficiency challenges, particularly in backward propagation, optimizer, and self-attention operations. To address these challenges, we propose SecP-Tuning, the first MPC-based framework designed for efficient, privacy-preserving prompt tuning of LLMs. SecP-Tuning innovatively integrates Forward-only Tuning (FoT) through the ``data owner-server interaction" paradigm, effectively removing the need for privacy-preserving computations in backward propagation and optimization processes. Furthermore, it devises an efficient privacy-preserving Random Feature Attention (RFA), effectively mitigating the computational complexity of softmax-based self-attention and circumventing MPC-incompatible nonlinear operations. Experimental results demonstrate that, compared to full-Parameter Supervised Fine-Tuning (SFT) and gradient-based prompt tuning, SecP-Tuning achieves approximately 12x and 16x end-to-end acceleration, as well as 17x and 20x reductions in communication overhead, respectively. Moreover, it delivers performance comparable to gradient-based methods across multiple few-shot tasks.

SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPC

TL;DR

SecP-Tuning is proposed, the first MPC-based framework designed for efficient, privacy-preserving prompt tuning of LLMs, and devises an efficient privacy-preserving Random Feature Attention (RFA), effectively mitigating the computational complexity of softmax-based self-attention and circumventing MPC-incompatible nonlinear operations.

Abstract

Large Language Models (LLMs) have revolutionized numerous fields, yet their adaptation to specialized tasks in privacy-sensitive domains such as healthcare and finance remains constrained due to the scarcity of accessible training data caused by stringent privacy requirements. Secure Multi-party Computation (MPC)-based privacy-preserving machine learning provides theoretical guarantees for the privacy of model parameters and data. However, its application to LLMs has been predominantly limited to inference, as fine-tuning introduces significant efficiency challenges, particularly in backward propagation, optimizer, and self-attention operations. To address these challenges, we propose SecP-Tuning, the first MPC-based framework designed for efficient, privacy-preserving prompt tuning of LLMs. SecP-Tuning innovatively integrates Forward-only Tuning (FoT) through the ``data owner-server interaction" paradigm, effectively removing the need for privacy-preserving computations in backward propagation and optimization processes. Furthermore, it devises an efficient privacy-preserving Random Feature Attention (RFA), effectively mitigating the computational complexity of softmax-based self-attention and circumventing MPC-incompatible nonlinear operations. Experimental results demonstrate that, compared to full-Parameter Supervised Fine-Tuning (SFT) and gradient-based prompt tuning, SecP-Tuning achieves approximately 12x and 16x end-to-end acceleration, as well as 17x and 20x reductions in communication overhead, respectively. Moreover, it delivers performance comparable to gradient-based methods across multiple few-shot tasks.

Paper Structure

This paper contains 39 sections, 13 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introction
  2. Related Work
  3. Preliminaries
  4. Softmax-based Self-Attention & Random Feature Attention
  5. Softmax-based Self-Attention.
  6. Random Feature Attention.
  7. Gradient-Free Optimization
  8. 2-out-of-2 Arithmetic Secret Sharing
  9. SecP-Tuning
  10. MPC-based Privacy-preserving Fine-tuning
  11. Privacy-preserving Forward-Only Tuning
  12. Privacy-preserving Random Feature Attention
  13. Experiments
  14. Setup
  15. MPC-Backend & Testbeds.
  16. ...and 24 more sections

Figures (3)

  • Figure 1: The time breakdown for SFT of RoBERTa$_{\text{LARGE}}$ (24 layers, 1024 dimensions) using MPC is analyzed with a sequence length of 512, along with a comparison of communication volumes across different sequence lengths.
  • Figure 2: Workflow of SecP-Tuning. SecP-Tuning leverages secure MPC to protect both training data and model parameters during fine-tuning. It addresses two key bottlenecks in PFT. First, it eliminates the computational overhead of backward and optimizer by adopting a FoT paradigm. Second, it improves the efficiency of privacy-preserving self attention by employing RFA.
  • Figure 3: Comparison of Time and Communication Overhead Between Privacy-Preserving RFA and Softmax-Based Privacy-Preserving Self-Attention.