Strategic Dialogue Assessment: The Crooked Path to Innocence

TL;DR

Strategic Dialogue Assessment (SDA) advances the study of non-cooperative language by integrating Gricean pragmatics with game-theoretic pragmatics to quantify how discourse moves manage commitments. By adapting the ME Game jury function and introducing a commitment-based taxonomy plus estimable proxies, SDA yields interpretable metrics—BaT, PaT, and NRBaT—that capture turn-level benefits and penalties and their cumulative effects. The Crooked Path Dataset (CPD), drawn from real courtroom cross-examinations, serves as a rigorous benchmark to validate SDA against human annotations and to assess LLMs' pragmatic understanding. Empirical results show SDA can predict outcome judgments and remains more robust to annotator subjectivity than existing metrics, though large language models still struggle with nuanced strategic reasoning, especially when explicit chain-of-thought reasoning is introduced. Overall, SDA provides a principled, empirically estimable framework for evaluating and guiding pragmatic reasoning in adversarial dialogue, with potential extensions to domains like political debates and negotiation.

Abstract

Language is often used strategically, particularly in high-stakes, adversarial settings, yet most work on pragmatics and LLMs centers on cooperativity. This leaves a gap in the systematic understanding of strategic communication in adversarial settings. To address this, we introduce SDA (Strategic Dialogue Assessment), a framework grounded in Gricean and game-theoretic pragmatics to assess strategic use of language. It adapts the ME Game jury function to make it empirically estimable for analyzing dialogue. Our approach incorporates two key adaptations: a commitment-based taxonomy of discourse moves, which provides a finer-grained account of strategic effects, and the use of estimable proxies grounded in Gricean maxims to operationalize abstract constructs such as credibility. Together, these adaptations build on discourse theory by treating discourse as the strategic management of commitments, enabling systematic evaluation of how conversational moves advance or undermine discourse goals. We further derive three interpretable metrics-Benefit at Turn (BAT), Penalty at Turn (PAT), and Normalized Relative Benefit at Turn (NRBAT)-to quantify the perceived strategic effects of discourse moves. We also present CPD (the Crooked Path Dataset), an annotated dataset of real courtroom cross-examinations, to demonstrate the framework's effectiveness. Using these tools, we evaluate a range of LLMs and show that LLMs generally exhibit limited pragmatic understanding of strategic language. While model size shows an increase in performance on our metrics, reasoning ability does not help and largely hurts, introducing overcomplication and internal confusion.

Figures (14)

• Figure 1: This figure shows a cross-examination dialogue, two idealized interpretations, and GPT-4o's judgment, which diverges from both interpretations and mistakenly treats the response as increasing the risk of losing. The full model output is shown in Appx. Fig. \ref{['gptfull']}.
• Figure 2: Examples of different commitment types and the corresponding values of PaT, BaT, and NRBaT
• Figure 3: An example illustrating the role of violations of maxims as diminishing the reliability (thus also the effect size) of the commitment.
• Figure 4: Annotation schema (guideline in Appx. \ref{['protocol']}).
• Figure 5: Distribution of Gricean maxim ratings and commitment types across trials