Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities

TL;DR

This work proposes a benchmark for assessing discourse processing in large language models by systematically probing three interrelated Implicit Causality biases: coreference, coherence, and anaphoric form. Through four experiments comparing mono- and multilingual LLMs of varied sizes against human baselines in German, the study shows that human-like IC coherence and form biases are largely absent in midrange models, with only the largest monolingual German BLOOM showing partial human-like coreference tendencies. The results emphasize a hierarchical pattern where robust coreference and coherence biases are prerequisites for anaphoric form biases, and suggest that current public, non-finetuned models struggle to generalize human discourse knowledge. The authors frame IC biases as a scalable, interpretable benchmark for discourse capabilities and highlight the need for larger, instruction-tuned models and cross-linguistic expansion to better approximate human discourse understanding in LLMs.

Abstract

In this paper, we compare data generated with mono- and multilingual LLMs spanning a range of model sizes with data provided by human participants in an experimental setting investigating well-established discourse biases. Beyond the comparison as such, we aim to develop a benchmark to assess the capabilities of LLMs with discourse biases as a robust proxy for more general discourse understanding capabilities. More specifically, we investigated Implicit Causality verbs, for which psycholinguistic research has found participants to display biases with regard to three phenomena:\ the establishment of (i) coreference relations (Experiment 1), (ii) coherence relations (Experiment 2), and (iii) the use of particular referring expressions (Experiments 3 and 4). With regard to coreference biases we found only the largest monolingual LLM (German Bloom 6.4B) to display more human-like biases. For coherence relation, no LLM displayed the explanation bias usually found for humans. For referring expressions, all LLMs displayed a preference for referring to subject arguments with simpler forms than to objects. However, no bias effect on referring expression was found, as opposed to recent studies investigating human biases.

Figures (8)

• Figure 1: Three-level model of discourse expectations associated with IC
• Figure 2: Individual I-Caus and I-Cons biases for stimulus-experiencer and experiencer-stimulus verbs from SolstadBott2022's SolstadBott2022 study. I-Caus bias is plotted on the $x$ axis and I-Cons bias on the $y$ axis (1 = coreference to the subject, 0 = coreference to the object) . The diamond-shaped points represent the average I-Caus and I-Cons biases for the two verb classes, respectively.
• Figure 3: I-Caus and I-Cons biases of individual verbs for human data (Figure \ref{['coref_humans']}) and the LLMs used in this study. I-Caus biases are plotted on the x axis and I-Cons biases on the y axis (1 = subject coreference, 0 = object coreference). The average I-Caus and I-Cons biases for stimulus-experiencer (stim.-exp.) and experiencer-stimulus (exp.-stim.) verbs are plotted as a diamond-shaped points.
• Figure 4: Distributions of discourse relations produced in continuations after prompts ending in a comma with stimulus-experiencer (SE) and experiencer-stimulus (ES) IC verbs.
• Figure 5: Coherence biases of stimulus-experiencer and experiencer-stimulus verbs (left and right-hand bars, respectively) for human data (Figure \ref{['coherence_humans']}) and the LLMs. The clearly predominant relation in the human data, Explanations, is always plotted as the bottom-most category in the bars.