Do Large Language Models (Really) Need Statistical Foundations?

TL;DR

This paper argues that large language models (LLMs) inherently operate as statistical systems due to their data-driven training and stochastic text generation, while their black-box complexity makes purely mechanistic theory impractical. It contends that statistical methods are not only useful but often essential for understanding and improving LLMs, particularly in handling uncertainty, evaluation, and data-driven design. The authors outline a mosaic of statistical research directions—ranging from alignment and uncertainty quantification to data mixture optimization, watermarking, and synthetic data generation—illustrating how statistics can guide both development and real-world deployment. They advocate a bottom-up, problem-driven evolution of the field, cautioning against waiting for a unifying theory and urging proactive engagement by the statistics community to shape the future of LLMs.

Abstract

Large language models (LLMs) represent a new paradigm for processing unstructured data, with applications across an unprecedented range of domains. In this paper, we address, through two arguments, whether the development and application of LLMs would genuinely benefit from foundational contributions from the statistics discipline. First, we argue affirmatively, beginning with the observation that LLMs are inherently statistical models due to their profound data dependency and stochastic generation processes, where statistical insights are naturally essential for handling variability and uncertainty. Second, we argue that the persistent black-box nature of LLMs -- stemming from their immense scale, architectural complexity, and development practices often prioritizing empirical performance over theoretical interpretability -- renders closed-form or purely mechanistic analyses generally intractable, thereby necessitating statistical approaches due to their flexibility and often demonstrated effectiveness. To substantiate these arguments, the paper outlines several research areas -- including alignment, watermarking, uncertainty quantification, evaluation, and data mixture optimization -- where statistical methodologies are critically needed and are already beginning to make valuable contributions. We conclude with a discussion suggesting that statistical research concerning LLMs will likely form a diverse ``mosaic'' of specialized topics rather than deriving from a single unifying theory, and highlighting the importance of timely engagement by our statistics community in LLM research.