Quantifying Noise in Language Generation
Aaron Li, Ian Zhang
TL;DR
This work analyzes how finite noise affects language generation in the limit, within the Kleinberg–Mullainathan framework. It shows a sharp separation between noiseless and noisy regimes for both uniform and non-uniform generation and proves that for any finite noise level $i\ge1$, generation with noise level $i$ is equivalent to generation with noise level $1$. It provides a simple, structural characterization of uniform noise-dependent generatability via the noisy-closure dimension $ ext{NC}_1(\mathcal{C})<\infty$, and extends analogous results to non-uniform settings, including a complete characterization of non-uniform noise-dependent generation. These results resolve open questions about the role of noisy labels and establish a unified view of finite-noise generation across uniform and non-uniform variants, with implications for robustness in language-generation systems.
Abstract
Kleinberg and Mullainathan recently proposed a formal framework for studying the phenomenon of language generation, called language generation in the limit. In this model, an adversary gives an enumeration of example strings from an unknown target language, and the algorithm is tasked with correctly generating unseen strings from the target language within finite time. Refined notions of non-uniform and uniform generation were later introduced by Li, Raman, and Tewari (2025), and a noisy model was introduced by Raman and Raman (2025), which allows the adversary to insert extraneous strings. A natural question in the noisy model is to quantify the effect of noise, by studying the impact of each additional extraneous string. We show two complementary results in this setting. We first show that for both uniform and non-uniform generation, a single noisy string strictly reduces the set of collections that can be generated, thus answering an open question in Raman and Raman (2025). Then, we show for both uniform and non-uniform generation that generation with a single noisy string is equivalent to generation with any finite amount of noise, sharply contrasting with the strict hierarchy for noisy generation in the limit shown by Bai, Panigrahi, and Zhang (2026). Finally, we leverage our previous results to provide the first known characterization for non-uniform noise-dependent generatability.