DiscoTrack: A Multilingual LLM Benchmark for Discourse Tracking

TL;DR

DiscoTrack introduces a multilingual benchmark spanning 12 languages to assess discourse tracking across salience, entity tracking, discourse relations, and bridging. By providing 11 subtasks grounded in long-range, multi-sentential reasoning, it reveals that current LLMs struggle with implicit discourse structure even in English and non-English contexts. The evaluation includes human performance, few-shot vs zero-shot analysis, and error analysis, highlighting gaps and language-specific patterns. The work offers an openly available resource to drive development in discourse-aware modeling and evaluation, with implications for multilingual NLP and long-document understanding.

Abstract

Recent LLM benchmarks have tested models on a range of phenomena, but are still focused primarily on natural language understanding for extraction of explicit information, such as QA or summarization, with responses often targeting information from individual sentences. We are still lacking more challenging, and importantly also multilingual, benchmarks focusing on implicit information and pragmatic inferences across larger documents in the context of discourse tracking: integrating and aggregating information across sentences, paragraphs and multiple speaker utterances. To this end, we present DiscoTrack, an LLM benchmark targeting a range of tasks across 12 languages and four levels of discourse understanding: salience recognition, entity tracking, discourse relations and bridging inference. Our evaluation shows that these tasks remain challenging, even for state-of-the-art models.

Figures (4)

• Figure 1: Sample discourse illustrating associative bridging anaphora, implicit cause and precedence discourse relations, tracking the entity Mary, and relative salience, where the less salient 'Jamie' is shown in gray.
• Figure 2: LLM and human performance (accuracy %).
• Figure 3: Confusion matrix for the relation classification task in the zero-shot setting.
• Figure 4: Confusion matrices for (a) discourse relation classification in the few-shot setting, (b) bridging type classification in the zero-shot setting, (c) bridging type classification in the few-shot setting, and (d) human performance on bridging type classification.