Quantifying non deterministic drift in large language models

TL;DR

The paper addresses the problem of baseline nondeterministic drift in large language models by measuring output variability under an operator-free regime across two public models, five prompt categories, two temperatures, and two deployment modes. It adopts a systematic, measurement-focused approach using lexical metrics such as unique output fraction and Jaccard similarity to establish a conservative baseline for drift, while discussing the potential for semantic metrics in future work. Key findings show that nondeterminism persists even at $T=0.0$ and intensifies with perturbations and higher temperatures, with distinct patterns dependent on model size and deployment. The work provides a variance-budget perspective to guide acceptable drift levels and highlights the need for semantic drift measures and longer-horizon, multi-agent evaluations to improve reliability of future LLM deployments.

Abstract

Large language models (LLMs) are widely used for tasks ranging from summarisation to decision support. In practice, identical prompts do not always produce identical outputs, even when temperature and other decoding parameters are fixed. In this work, we conduct repeated-run experiments to empirically quantify baseline behavioural drift, defined as output variability observed when the same prompt is issued multiple times under operator-free conditions. We evaluate two publicly accessible models, gpt-4o-mini and llama3.1-8b, across five prompt categories using exact repeats, perturbed inputs, and reuse modes at temperatures of 0.0 and 0.7. Drift is measured using unique output fractions, lexical similarity, and word count statistics, enabling direct comparison across models, prompting modes, and deployment types. The results show that nondeterminism persists even at temperature 0.0, with distinct variability patterns by model size, deployment, and prompt type. We situate these findings within existing work on concept drift, behavioural drift, and infrastructure-induced nondeterminism, discuss the limitations of lexical metrics, and highlight emerging semantic approaches. By establishing a systematic empirical baseline in the absence of stabilisation techniques, this study provides a reference point for evaluating future drift mitigation and control methods.

Figures (3)

• Figure 1: Mean unique output fraction for exact repeats at temperature 0.0
• Figure 2: Mean average Jaccard similarity for exact repeats at temperature 0.0
• Figure 3: Unique output fraction distribution by prompt for exact repeats at temperature 0.0