Uncertainty in Natural Language Generation: From Theory to Applications

TL;DR

<3-5 sentence high-level summary> The paper addresses uncertainty in natural language generation (NLG) and argues that principled uncertainty representations can enhance trust, inclusivity, and adaptability in NLG systems. It builds a formal foundation using a possible-worlds framework, probability measures, and multiple interpretations, and then introduces a two-dimensional taxonomy that separates data- and model-related sources of uncertainty beyond the aleatoric/epistemic dichotomy. The authors describe practical applications across decoding, controllable generation, self-assessment, selective answering, calibration, conformal prediction, Bayesian inference, verbalised uncertainty, and active learning, demonstrating how disentangled uncertainty can power robust NLG. The work aims to guide the development of more trustworthy, diverse, and user-aligned language interfaces through principled uncertainty modeling and evaluation.

Abstract

Recent advances of powerful Language Models have allowed Natural Language Generation (NLG) to emerge as an important technology that can not only perform traditional tasks like summarisation or translation, but also serve as a natural language interface to a variety of applications. As such, it is crucial that NLG systems are trustworthy and reliable, for example by indicating when they are likely to be wrong; and supporting multiple views, backgrounds and writing styles -- reflecting diverse human sub-populations. In this paper, we argue that a principled treatment of uncertainty can assist in creating systems and evaluation protocols better aligned with these goals. We first present the fundamental theory, frameworks and vocabulary required to represent uncertainty. We then characterise the main sources of uncertainty in NLG from a linguistic perspective, and propose a two-dimensional taxonomy that is more informative and faithful than the popular aleatoric/epistemic dichotomy. Finally, we move from theory to applications and highlight exciting research directions that exploit uncertainty to power decoding, controllable generation, self-assessment, selective answering, active learning and more.

Figures (2)

• Figure 1: We propose the double triangle of language production, extending the traditional "Triangle of Reference" ogden-richards-1923meaning to NLG: a schematic diagram of the key aspects involved when a speaker produces a linguistic signal (output) given a prompt (input) in the context of a communicative task. The blue arrows correspond to sub-processes where one-to-many mappings may arise, thus leading to possible variation, which gives rise to uncertainty.
• Figure 2: The main sources of uncertainty in NLG relate to the nature of the data and the tools available to the modeller. We depict them in red and blue (data and model respectively). Being able to reduce uncertainty is hardly determined by the sources themselves, rather, reducibility depends on decisions (from data collection to model design and training) by the NLG practitioner. For each source, we list along the vertical axis some considerations that affect reducibility. This view generalises the traditional aleatoric vs. epsitemic categorisation of uncertainty.