Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Private LoRA Fine-tuning of Open-Source LLMs with Homomorphic Encryption

Jordan Frery, Roman Bredehoft, Jakub Klemsa, Arthur Meyre, Andrei Stoian

TL;DR

This work presents a privacy-preserving protocol for fine-tuning open-source LLMs using LoRA in conjunction with homomorphic encryption. The data owner coordinates LoRA updates while the server, operating on encrypted activations, handles the heavy linear computations with W, enabling secure private adaptation. The authors demonstrate feasibility on Llama-3.2-1B with HE-compatible quantization, achieving convergence close to FP32 and reporting a heavy but scalable server-side cost (approximately 216 seconds per token for a training step) while keeping client-side computation minimal (around 0.025 MFLOP/s). The approach enables confidential tasks such as knowledge-base QA, private code assistants, and sensitive-domain analysis, and shows clear potential for scaling through multi-server orchestration.

Abstract

Preserving data confidentiality during the fine-tuning of open-source Large Language Models (LLMs) is crucial for sensitive applications. This work introduces an interactive protocol adapting the Low-Rank Adaptation (LoRA) technique for private fine-tuning. Homomorphic Encryption (HE) protects the confidentiality of training data and gradients handled by remote worker nodes performing the bulk of computations involving the base model weights. The data owner orchestrates training, requiring minimal local computing power and memory, thus alleviating the need for expensive client-side GPUs. We demonstrate feasibility by fine-tuning a Llama-3.2-1B model, presenting convergence results using HE-compatible quantization and performance benchmarks for HE computations on GPU hardware. This approach enables applications such as confidential knowledge base question answering, private codebase fine-tuning for AI code assistants, AI agents for drafting emails based on a company's email archive, and adapting models to analyze sensitive legal or healthcare documents.

Private LoRA Fine-tuning of Open-Source LLMs with Homomorphic Encryption

TL;DR

This work presents a privacy-preserving protocol for fine-tuning open-source LLMs using LoRA in conjunction with homomorphic encryption. The data owner coordinates LoRA updates while the server, operating on encrypted activations, handles the heavy linear computations with W, enabling secure private adaptation. The authors demonstrate feasibility on Llama-3.2-1B with HE-compatible quantization, achieving convergence close to FP32 and reporting a heavy but scalable server-side cost (approximately 216 seconds per token for a training step) while keeping client-side computation minimal (around 0.025 MFLOP/s). The approach enables confidential tasks such as knowledge-base QA, private code assistants, and sensitive-domain analysis, and shows clear potential for scaling through multi-server orchestration.

Abstract

Preserving data confidentiality during the fine-tuning of open-source Large Language Models (LLMs) is crucial for sensitive applications. This work introduces an interactive protocol adapting the Low-Rank Adaptation (LoRA) technique for private fine-tuning. Homomorphic Encryption (HE) protects the confidentiality of training data and gradients handled by remote worker nodes performing the bulk of computations involving the base model weights. The data owner orchestrates training, requiring minimal local computing power and memory, thus alleviating the need for expensive client-side GPUs. We demonstrate feasibility by fine-tuning a Llama-3.2-1B model, presenting convergence results using HE-compatible quantization and performance benchmarks for HE computations on GPU hardware. This approach enables applications such as confidential knowledge base question answering, private codebase fine-tuning for AI code assistants, AI agents for drafting emails based on a company's email archive, and adapting models to analyze sensitive legal or healthcare documents.
Paper Structure (45 sections, 11 equations, 4 figures, 4 tables)

This paper contains 45 sections, 11 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Prior Work
  3. Low-Rank Adaptation
  4. Split Edge-Cloud LLM Fine-tuning
  5. Non-interactive Encrypted Training
  6. Preliminaries
  7. Homomorphic Encryption Scheme
  8. RLWE Ciphertexts.
  9. LWE Ciphertexts.
  10. Mask Generation via PRNG.
  11. Key HE Operations
  12. Method
  13. LoRA Fine-tuning
  14. Quantization
  15. Challenges with Standard LLM Quantization in HE.
  16. ...and 30 more sections

Figures (4)

  • Figure 1: Private LoRA fine-tuning computation split: Client handles LoRA weights $U, D$ and non-linearities; Server handles base model weights $W$ under HE.
  • Figure 2: Training-loss trajectories for quantization settings in Table \ref{['tab:quant_settings_overview']} on Orca Math.
  • Figure 3: Llama-3.2-1B: training-loss comparison between 8-bit quantized cleartext (blue) and HE execution (red) over the first five steps.
  • Figure 4: Bit error rate versus bit position for homomorphic dot products. The y-axis shows the input vector dimension $d_{in}$ (dimensions tested: 768, 2048, 8192). The x-axis shows the bit position (MSB near 21, left; LSB=0, right). All computations used the fixed polynomial size $N=2048$. Higher input dimensions increase LSB error due to noise accumulation, but high-order bits (positions 12 and higher) exhibit error rates under 1%, preserving $\gamma=12$ MSBs reliably.