Towards an Analysis of Discourse and Interactional Pragmatic Reasoning Capabilities of Large Language Models

TL;DR

The paper addresses the lack of a cohesive framework for evaluating pragmatic reasoning in Large Language Models by introducing a twofold taxonomy that separates discourse pragmatics from interactional pragmatics. It surveys existing tests and methodologies across these domains, highlighting notable work on discourse phenomena (e.g., implicatures, scalar inferences, metaphor) and the relative scarcity of standardized approaches for interactional pragmatics. The authors provide a structured contribution: a precise distinction between the two pragmatic categories and a synthesis of tested phenomena, grounding relationships, methodologies, and model coverage. This work aims to standardize evaluation of pragmatic grounding in LLMs and guide future benchmark design and research directions.

Abstract

In this work, we want to give an overview on which pragmatic abilities have been tested in LLMs so far and how these tests have been carried out. To do this, we first discuss the scope of the field of pragmatics and suggest a subdivision into discourse pragmatics and interactional pragmatics. We give a non-exhaustive overview of the phenomena of those two subdomains and the methods traditionally used to analyze them. We subsequently consider the resulting heterogeneous set of phenomena and methods as a starting point for our survey of work on discourse pragmatics and interactional pragmatics in the context of LLMs.