Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VeriLoRA: Fine-Tuning Large Language Models with Verifiable Security via Zero-Knowledge Proofs

Guofu Liao, Taotao Wang, Shengli Zhang, Jiqun Zhang, Shi Long, Dacheng Tao

TL;DR

VeriLoRA introduces the first end-to-end zero-knowledge verification framework for LoRA-based fine-tuning of large language models, enabling secure and private verification of forward pass, backward pass, and parameter updates. It combines arithmetic-verification via sumcheck and multilinear extensions with non-arithmetic protocols built on lookup arguments to handle activations, Softmax, and tensor operations, all supported by Hyrax-style polynomial commitments. The approach is validated on open-source LLMs up to 13B parameters using GPU-accelerated implementations, showing practical verification times and substantial overhead for proof generation, but manageable verification times and strong privacy guarantees. This work bridges cryptography and ML by enabling trustworthy fine-tuning in untrusted or sensitive environments and sets the stage for future lighter-weight verifiable training systems.

Abstract

Fine-tuning large language models (LLMs) is crucial for adapting them to specific tasks, yet it remains computationally demanding and raises concerns about correctness and privacy, particularly in untrusted environments. Although parameter-efficient methods like Low-Rank Adaptation (LoRA) significantly reduce resource requirements, ensuring the security and verifiability of fine-tuning under zero-knowledge constraints remains an unresolved challenge. To address this, we introduce VeriLoRA, the first framework to integrate LoRA fine-tuning with zero-knowledge proofs (ZKPs), achieving provable security and correctness. VeriLoRA employs advanced cryptographic techniques -- such as lookup arguments, sumcheck protocols, and polynomial commitments -- to verify both arithmetic and non-arithmetic operations in Transformer-based architectures. The framework provides end-to-end verifiability for forward propagation, backward propagation, and parameter updates during LoRA fine-tuning, while safeguarding the privacy of model parameters and training data. Leveraging GPU-based implementations, VeriLoRA demonstrates practicality and efficiency through experimental validation on open-source LLMs like LLaMA, scaling up to 13 billion parameters. By combining parameter-efficient fine-tuning with ZKPs, VeriLoRA bridges a critical gap, enabling secure and trustworthy deployment of LLMs in sensitive or untrusted environments.

VeriLoRA: Fine-Tuning Large Language Models with Verifiable Security via Zero-Knowledge Proofs

TL;DR

VeriLoRA introduces the first end-to-end zero-knowledge verification framework for LoRA-based fine-tuning of large language models, enabling secure and private verification of forward pass, backward pass, and parameter updates. It combines arithmetic-verification via sumcheck and multilinear extensions with non-arithmetic protocols built on lookup arguments to handle activations, Softmax, and tensor operations, all supported by Hyrax-style polynomial commitments. The approach is validated on open-source LLMs up to 13B parameters using GPU-accelerated implementations, showing practical verification times and substantial overhead for proof generation, but manageable verification times and strong privacy guarantees. This work bridges cryptography and ML by enabling trustworthy fine-tuning in untrusted or sensitive environments and sets the stage for future lighter-weight verifiable training systems.

Abstract

Fine-tuning large language models (LLMs) is crucial for adapting them to specific tasks, yet it remains computationally demanding and raises concerns about correctness and privacy, particularly in untrusted environments. Although parameter-efficient methods like Low-Rank Adaptation (LoRA) significantly reduce resource requirements, ensuring the security and verifiability of fine-tuning under zero-knowledge constraints remains an unresolved challenge. To address this, we introduce VeriLoRA, the first framework to integrate LoRA fine-tuning with zero-knowledge proofs (ZKPs), achieving provable security and correctness. VeriLoRA employs advanced cryptographic techniques -- such as lookup arguments, sumcheck protocols, and polynomial commitments -- to verify both arithmetic and non-arithmetic operations in Transformer-based architectures. The framework provides end-to-end verifiability for forward propagation, backward propagation, and parameter updates during LoRA fine-tuning, while safeguarding the privacy of model parameters and training data. Leveraging GPU-based implementations, VeriLoRA demonstrates practicality and efficiency through experimental validation on open-source LLMs like LLaMA, scaling up to 13 billion parameters. By combining parameter-efficient fine-tuning with ZKPs, VeriLoRA bridges a critical gap, enabling secure and trustworthy deployment of LLMs in sensitive or untrusted environments.

Paper Structure

This paper contains 41 sections, 21 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Proofs, Arguments and Polynomial Commitments
  4. Proofs and Arguments
  5. Polynomial Commitments
  6. Sumcheck, and MLE
  7. Sumcheck Protocol
  8. Multilinear Extensions
  9. Verifiable Tensor Operation
  10. Lookup Arguments for Non-arithmetic Operations
  11. Gradient Computations for LoRA in LLMs
  12. Structure of LLM
  13. Transformer Layers
  14. Final Output Layer
  15. LoRA Fine Tuning
  16. ...and 26 more sections

Figures (2)

  • Figure 1: The typical structure of an LLM with Transformer layers, and the entire computation procedure of LoRA fine-tuning applied to an LLM.
  • Figure 2: The proving time of VeriLoRA for different models (in seconds): (a) the forward propagation phase; (b) the backward propagation phase; (c) the parameter update phase, (d) the total.

Theorems & Definitions (2)

  • Definition 2.1: Interactive Proof System
  • Definition 2.2: Zero-Knowledge Proof