Is Information Density Uniform when Utterances are Grounded on Perception and Discourse?

TL;DR

This work tests the Uniform Information Density hypothesis in ecologically plausible, multimodal settings by grounding utterances on visual perception and discourse. Using multilingual vision–language models, it shows that grounding consistently smooths information distribution across words and across extended visual narratives, with stronger effects at unit onsets. The study combines two datasets (image–caption and visual storytelling) across 33 languages and demonstrates a typologically robust trend: visual context reduces surprisal variance, while discourse context adds further smoothing, producing a front-loaded UID profile. The findings advance a context-sensitive UID account and point to future work integrating dynamic perception, more nuanced modality decompositions, and human validation.

Abstract

The Uniform Information Density (UID) hypothesis posits that speakers are subject to a communicative pressure to distribute information evenly within utterances, minimising surprisal variance. While this hypothesis has been tested empirically, prior studies are limited exclusively to text-only inputs, abstracting away from the perceptual context in which utterances are produced. In this work, we present the first computational study of UID in visually grounded settings. We estimate surprisal using multilingual vision-and-language models over image-caption data in 30 languages and visual storytelling data in 13 languages, together spanning 11 families. We find that grounding on perception consistently smooths the distribution of information, increasing both global and local uniformity across typologically diverse languages compared to text-only settings. In visual narratives, grounding in both image and discourse contexts has additional effects, with the strongest surprisal reductions occurring at the onset of discourse units. Overall, this study takes a first step towards modelling the temporal dynamics of information flow in ecologically plausible, multimodal language use, and finds that grounded language exhibits greater information uniformity, supporting a context-sensitive formulation of UID.

Figures (19)

• Figure 1: Global and local UID values under the U condition (caption in isolation) and P condition (caption with visual context) across the 30 typologically diverse languages, from 10 language families, in Ground-XM3600 dataset. Points below the dashed diagonal line indicate increased uniformity (i.e., decreased global or local variability) when the caption's surprisal is conditioned on the image context.
• Figure 2: Kernel density estimation of global UID values for captions in Ground-XM3600, conditioned either on their respective image (P) or without any contextual image (U). Curves are truncated at the 99th percentile to enhance visual clarity.
• Figure 3: Each box represents the distribution of paragraph-level $\mathrm{UID}_{v}\xspace$ values across languages under four conditions: no context (U); paired image as context (P); all preceding paragraphs as context (D); all preceding paragraphs and interleaved images as context ($\text{P\xspace}+\text{D\xspace}$). Dashed horizontal lines indicate the mean $\mathrm{UID}_{v}\xspace$ per condition, averaged across all languages.
• Figure 4: Paragraph-level UID values averaged across BloomVIST stories (truncated at 20 paragraphs) under the utterance only (U), perceptual context (P), discourse context (D), and multimodal context ($\text{P\xspace}+\text{D\xspace}$) conditions.
• Figure 5: Average change in variance contribution ($\overline{\Delta C}_{\text{POS}}$) by POS across languages for UID reduction failure cases in Ground-XM3600. Rows are ordered, from top to bottom, by descending cross-linguistic mean. Intuitively, red cells denote an increased divergence from the sequence mean under visual grounding (P) relative to text-only (U) conditions.