Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Token-Efficient Change Detection in LLM APIs

Timothée Chauvin, Clément Lalanne, Erwan Le Merrer, Jean-Michel Loubes, François Taïani, Gilles Tredan

TL;DR

This work tackles the problem of detecting changes in LLM APIs without access to model weights or log-probabilities. It introduces Border Input Tracking (B3IT), which exploits border inputs—where multiple top logits tie at low temperature—to achieve highly sensitive change detection under a strict black-box setting. The authors develop a Local Asymptotic Normality framework that ties detectability to a data-dependent SNR, and prove a phase transition: at zero temperature, border inputs yield maximal detectability when at least two top tokens tie. Empirical validation on TinyChange and 93 commercial endpoints shows B3IT achieves performance approaching grey-box methods at roughly 1/30th the cost of the best black-box baselines, highlighting its practical utility for continuous LLM API monitoring.

Abstract

Remote change detection in LLMs is a difficult problem. Existing methods are either too expensive for deployment at scale, or require initial white-box access to model weights or grey-box access to log probabilities. We aim to achieve both low cost and strict black-box operation, observing only output tokens. Our approach hinges on specific inputs we call Border Inputs, for which there exists more than one output top token. From a statistical perspective, optimal change detection depends on the model's Jacobian and the Fisher information of the output distribution. Analyzing these quantities in low-temperature regimes shows that border inputs enable powerful change detection tests. Building on this insight, we propose the Black-Box Border Input Tracking (B3IT) scheme. Extensive in-vivo and in-vitro experiments show that border inputs are easily found for non-reasoning tested endpoints, and achieve performance on par with the best available grey-box approaches. B3IT reduces costs by $30\times$ compared to existing methods, while operating in a strict black-box setting.

Token-Efficient Change Detection in LLM APIs

TL;DR

This work tackles the problem of detecting changes in LLM APIs without access to model weights or log-probabilities. It introduces Border Input Tracking (B3IT), which exploits border inputs—where multiple top logits tie at low temperature—to achieve highly sensitive change detection under a strict black-box setting. The authors develop a Local Asymptotic Normality framework that ties detectability to a data-dependent SNR, and prove a phase transition: at zero temperature, border inputs yield maximal detectability when at least two top tokens tie. Empirical validation on TinyChange and 93 commercial endpoints shows B3IT achieves performance approaching grey-box methods at roughly 1/30th the cost of the best black-box baselines, highlighting its practical utility for continuous LLM API monitoring.

Abstract

Remote change detection in LLMs is a difficult problem. Existing methods are either too expensive for deployment at scale, or require initial white-box access to model weights or grey-box access to log probabilities. We aim to achieve both low cost and strict black-box operation, observing only output tokens. Our approach hinges on specific inputs we call Border Inputs, for which there exists more than one output top token. From a statistical perspective, optimal change detection depends on the model's Jacobian and the Fisher information of the output distribution. Analyzing these quantities in low-temperature regimes shows that border inputs enable powerful change detection tests. Building on this insight, we propose the Black-Box Border Input Tracking (B3IT) scheme. Extensive in-vivo and in-vitro experiments show that border inputs are easily found for non-reasoning tested endpoints, and achieve performance on par with the best available grey-box approaches. B3IT reduces costs by compared to existing methods, while operating in a strict black-box setting.
Paper Structure (72 sections, 6 theorems, 102 equations, 7 figures, 2 algorithms)

This paper contains 72 sections, 6 theorems, 102 equations, 7 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions.
  3. System model
  4. Change detection.
  5. Types of changes.
  6. Single-token perspective.
  7. Theoretical analysis and guidelines
  8. Rationale
  9. The Local Asymptotic Normality Framework
  10. Takeaway.
  11. Decomposition for LLMs: exploiting the last layer
  12. The Zero-Temperature Limit
  13. Takeaway.
  14. B3IT: Black-Box Border Input Tracking
  15. The B3IT Initialization Stage
  16. ...and 57 more sections

Key Result

Theorem 3.1

Let $\alpha \in (0, 1)$. If $(\phi_n)$ is a sequence of tests such that, for every $n$, $\phi_n$ is the test with the lowest Type-II error among all tests with Type-I error at most $\alpha$ in testing $\mathbf{p}_0$ vs $\mathbf{p}_n$, then where $\mathrm{SNR}^2(h):= h^T(J^T F (\mathbf{p}_0)^{-1} J) h$, and $\text{Type-II}(\phi_n)$ refers to the error of $\phi_n$ under $\mathbf{p}_n$, and where $Q

Figures (7)

  • Figure 1: Performance increases with cost. We select 5 prompts and 3 samples/prompt for subsequent experiments.
  • Figure 2: B3IT outperforms all black-box methods by a wide margin, and approaches the performance of the grey-box LT method.
  • Figure 3: Detection performance on the TinyChange fine-tuning difficulty scale.
  • Figure 4: Empirical CDF of requests per BI across endpoints, for various temperatures.
  • Figure 5: Detected changes on endpoints over 23 days.
  • ...and 2 more figures

Theorems & Definitions (13)

  • Theorem 3.1: Optimal Tests in the LAN Regime
  • proof
  • Lemma 3.2: $\mathrm{SNR}^2(h)$ in LLMs
  • proof
  • Theorem 3.3: Phase Transition
  • proof
  • Remark 3.4: On the condition $h^T \left(J_z^T\Sigma_{\mathcal{M}}J_z\right) h \neq 0$
  • Theorem 4.1: Type-I error
  • proof
  • Theorem 4.2: Type-II error
  • ...and 3 more