Language Generation with Infinite Contamination
Anay Mehrotra, Grigoris Velegkas, Xifan Yu, Felix Zhou
TL;DR
This work develops a comprehensive theory of language generation in the limit under contamination, extending the original generation framework to realistic noisy data with omissions. It introduces two general algorithmic templates—Finite-Expansion Sub-Routine and Priority-Based Intersection—and analyzes generation both with and without density under diverse contamination regimes, including finite, vanishing, and constant noise, as well as arbitrary omissions. A key finding is that generation in the limit tolerates infinite contamination as long as the noise fraction vanishes, whereas dense (set- and element-based) generation is substantially less robust, with precise impossibility results under finite or infinite contamination. To sharpen practical relevance, the authors introduce a beyond-worst-case bounded-adversary model inspired by curriculum learning and show that dense generation becomes achievable under infinite contamination when the adversary’s presentation order respects a bounded-displacement constraint, linking curriculum learning to robust web-data learning. They also derive concrete results for membership-oracle access under finite contamination and establish a rich set of lower bounds and reductions between set-based and element-based density notions, highlighting the trade-offs between breadth, density, and contamination in formal learning-from-data settings.
Abstract
We study language generation in the limit, where an algorithm observes an adversarial enumeration of strings from an unknown target language $K$ and must eventually generate new, unseen strings from $K$. Kleinberg and Mullainathan [KM24] proved that generation is achievable in surprisingly general settings. But their generator suffers from ``mode collapse,'' producing from an ever-smaller subset of the target. To address this, Kleinberg and Wei [KW25] require the generator's output to be ``dense'' in the target language. They showed that generation with density, surprisingly, remains achievable at the same generality. Both results assume perfect data: no noisy insertions and no omissions. This raises a central question: how much contamination can generation tolerate? Recent works made partial progress on this question by studying (non-dense) generation with either finite amounts of noise (but no omissions) or omissions (but no noise). We characterize robustness under contaminated enumerations: 1. Generation under Contamination: Language generation in the limit is achievable for all countable collections iff the fraction of contaminated examples converges to zero. When this fails, we characterize which collections are generable. 2. Dense Generation under Contamination: Dense generation is strictly less robust to contamination than generation. As a byproduct, we resolve an open question of Raman and Raman [ICML25] by showing that generation is possible with only membership oracle access under finitely many contaminated examples. Finally, we introduce a beyond-worst-case model inspired by curriculum learning and prove that dense generation is achievable even with infinite contamination provided the fraction of contaminated examples converges to zero. This suggests curriculum learning may be crucial for learning from noisy web data.