Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation

Yanpei Guo, Wenjie Qu, Linyu Wu, Shengfang Zhai, Lionel Z. Wang, Ming Xu, Yue Liu, Binhang Yuan, Dawn Song, Jiaheng Zhang

TL;DR

IMMACULATE is a practical auditing framework that detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling-without trusted hardware or access to model internals, achieving strong detection guarantees while amortizing cryptographic overhead.

Abstract

Commercial large language models are typically deployed as black-box API services, requiring users to trust providers to execute inference correctly and report token usage honestly. We present IMMACULATE, a practical auditing framework that detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling-without trusted hardware or access to model internals. IMMACULATE selectively audits a small fraction of requests using verifiable computation, achieving strong detection guarantees while amortizing cryptographic overhead. Experiments on dense and MoE models show that IMMACULATE reliably distinguishes benign and malicious executions with under 1% throughput overhead. Our code is published at https://github.com/guo-yanpei/Immaculate.

IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation

TL;DR

IMMACULATE is a practical auditing framework that detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling-without trusted hardware or access to model internals, achieving strong detection guarantees while amortizing cryptographic overhead.

Abstract

Commercial large language models are typically deployed as black-box API services, requiring users to trust providers to execute inference correctly and report token usage honestly. We present IMMACULATE, a practical auditing framework that detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling-without trusted hardware or access to model internals. IMMACULATE selectively audits a small fraction of requests using verifiable computation, achieving strong detection guarantees while amortizing cryptographic overhead. Experiments on dense and MoE models show that IMMACULATE reliably distinguishes benign and malicious executions with under 1% throughput overhead. Our code is published at https://github.com/guo-yanpei/Immaculate.
Paper Structure (57 sections, 4 theorems, 22 equations, 7 figures, 5 tables, 3 algorithms)

This paper contains 57 sections, 4 theorems, 22 equations, 7 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Our approach.
  3. Preliminary and Related Work
  4. Notation.
  5. Cryptographic Commitment
  6. Verifiable Computation
  7. Related Works
  8. Threat Model
  9. System setting.
  10. Threat model.
  11. Auditing goal.
  12. Logit Distance Distribution
  13. Hybrid Computation Model: LLM Abstraction
  14. Hybrid computation model.
  15. Ideal and approximate execution.
  16. ...and 42 more sections

Key Result

Proposition 4.2

Model substitution introduces a systematic bias in logit outputs. As a result, model substitution typically yields substantially larger LDDs.

Figures (7)

  • Figure 1: The auditor sends random requests like other users and requires the model owner to prove the responses after they are received.
  • Figure 2: When fixing discrete selections ${d_i}$, the entire inference (green workflow) can be viewed as a continuous computation, as discrete selection no longer introduces branching uncertainty.
  • Figure 3: Logit TV-distance distribution of LLaMA3-70B. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.
  • Figure 4: Global logit TV-distance distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.
  • Figure 5: Global logit KL divergence distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.
  • ...and 2 more figures

Theorems & Definitions (8)

  • Definition 4.1: Logit Distance Distribution
  • Proposition 4.2
  • Proposition 4.3
  • Proposition 4.4
  • proof
  • proof
  • Proposition 6.1: Logit Commitment Optimality
  • proof