Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions

TL;DR

This paper conducts the first large-scale, statistically powered audit of native probabilistic sampling in frontier LLMs across 15 distributions and 11 models. It introduces a dual-protocol evaluation (Batch Generation vs Independent Requests) to separate long-context correction from true independent sampling and uses $\mathcal{W}_1$, KL, KS, and $\chi^2$ metrics to quantify fidelity against high-precision references. The key findings show a severe protocol-dependent gap: batch sampling achieves modest validity (median pass ~13%), while independent sampling nearly collapses (10 of 11 models pass none), with fidelity deteriorating as distributional complexity and sampling horizon $N$ increase. Downstream tests reveal tangible propagation of sampling failures into MCQ positional bias and demographic distribution violations in prompt generation, indicating that current LLMs lack a functional internal sampler and that external tools are necessary for statistical guarantees in stochastic tasks.

Abstract

As large language models (LLMs) transition from chat interfaces to integral components of stochastic pipelines across domains like educational assessment and synthetic data construction, the ability to faithfully sample from specified probability distributions has become a functional requirement rather than a theoretical curiosity. We present the first large-scale, statistically powered audit of native probabilistic sampling in frontier LLMs, benchmarking 11 models across 15 distributions. To disentangle failure modes, we employ a dual-protocol design: Batch Generation, where a model produces N=1000 samples within one response, and Independent Requests, comprising $N=1000$ stateless calls. We observe a sharp protocol asymmetry: batch generation achieves only modest statistical validity, with a 13% median pass rate, while independent requests collapse almost entirely, with 10 of 11 models passing none of the distributions. Beyond this asymmetry, we reveal that sampling fidelity degrades monotonically with distributional complexity and aggravates as the requested sampling horizon N increases. Finally, we demonstrate the propagation of these failures into downstream tasks: models fail to enforce uniform answer-position constraints in MCQ generation and systematically violate demographic targets in attribute-constrained text-to-image prompt synthesis. These findings indicate that current LLMs lack a functional internal sampler, necessitating the use of external tools for applications requiring statistical guarantees.

Figures (3)

• Figure 1: Overview of the Evaluation Pipeline. We systematically benchmark 11 frontier LLMs across 15 probability distributions spanning three complexity tiers. The evaluation employs a dual-protocol design to disentangle failure modes: Protocol A (Batch) produces samples sequentially within a single response, while Protocol B (Independent) produces samples via stateless single-sample calls. Distributional fidelity is rigorously quantified using statistical validity tests (KS, $\chi^2$) and geometric metrics ($\mathcal{W}_1$) against high-precision numpy reference samples.
• Figure 2: Distribution Complexity vs. Sampling Fidelity. (a) Statistical test pass rate decreases as distribution complexity increases from Tier I (Fundamental Priors) to Tier III (Heavy-Tailed & Complex). (b) Mean Wasserstein distance $\mathcal{W}_1$ increases with complexity, indicating poorer distributional fit.
• Figure 3: Effect of sample size ($N$) on DeepSeek-V3.2's Gaussian sampling quality. The dashed line in the KS test indicates the significance threshold $\alpha=0.01$.