Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse

Abstract

The Uniform Information Density (UID) hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. Of course, information rate in texts and discourses is not perfectly uniform. While these fluctuations can be viewed as theoretically uninteresting noise on top of a uniform target, another explanation is that UID is not the only functional pressure regulating information content in a language. Speakers may also seek to maintain interest, adhere to writing conventions, and build compelling arguments. In this paper, we propose one such functional pressure; namely that speakers modulate information rate based on location within a hierarchically-structured model of discourse. We term this the Structured Context Hypothesis and test it by predicting the surprisal contours of naturally occurring discourses extracted from large language models using predictors derived from discourse structure. We find that hierarchical predictors are significant predictors of a discourse's information contour and that deeply nested hierarchical predictors are more predictive than shallow ones. This work takes an initial step beyond UID to propose testable hypotheses for why the information rate fluctuates in predictable ways

Figures (7)

• Figure 1: Information contour of the wsj_1111 document from the English RST Discourse Treebank.
• Figure 2: Discourse sub-tree for a sentence in wsj_1111 from the English RST Discourse Treebank.
• Figure 3: Illustration of the pushes and pops from different parsing strategies, with top-down parsing (left) popping nodes in preorder, and bottom-up parsing (right) popping nodes in postorder. Note that the pushing of rules during the depth-first-search is equal in both cases.
• Figure 4: $\Delta\text{MSE}$ comparison of models trained on four RST-based predictor groups. Note that the scale for surprisal with a rolling window of 5 is smaller, as rolling average dependent variables exhibit less variance. All these results are statistically significant against the baseline ($p<0.001$).
• Figure 5: $\Delta\text{MSE}$ across dependent variables of all RST and Prose Structure (PS) predictors on the English data.