Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom

TL;DR

This work introduces SwordsmanImp, the first Chinese multi-turn implicature dataset sourced from a popular sitcom, annotated for Gricean maxims and used to evaluate LLMs on both multiple-choice implicature questions and open-ended explanations. In two experiments, GPT-4 attains near-human accuracy on MCQ tasks (≈94%), while most models struggle to produce reasonable, coherent explanations despite fluent text. The findings highlight a gap between surface-level MCQ performance and genuine explanatory ability, and show no strong dependence on which Gricean maxim is violated. The dataset provides a valuable benchmark for studying Chinese pragmatic reasoning in real-world dialog contexts and motivates further work on cross-linguistic and multimodal pragmatic evaluation.

Abstract

Understanding the non-literal meaning of an utterance is critical for large language models (LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourced from dialogues in the Chinese sitcom $\textit{My Own Swordsman}$. It includes 200 carefully handcrafted questions, all annotated on which Gricean maxims have been violated. We test eight close-source and open-source LLMs under two tasks: a multiple-choice question task and an implicature explanation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT-3.5 and several open-source models, demonstrate a lower accuracy ranging from 20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation of the implicatures generated by LLMs on their reasonability, logic and fluency. While all models generate largely fluent and self-consistent text, their explanations score low on reasonability except for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of the implicatures in the conversation. Moreover, we find LLMs' performance does not vary significantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derived from different maxims differently. Our data and code are available at https://github.com/sjtu-compling/llm-pragmatics.

Figures (5)

• Figure 1: An example entry in our dataset. The tick (✓) and fork (✗) denotes if the target sentence, marked in red, comply with or violate the sub-maxim. This entry belongs to the classes of Quality, Quantity and Manner. "sneak out" and "skate" translate to the same Chinese character "溜".
• Figure 2: Performance of models across question types in the multiple choice scenario. Dashed line represents chance accuracy
• Figure 3: Frequency of each option in models' answers.
• Figure 4: Models' interpretations to a character line (highlighted with red color) that carries extra meanings in a dialogue with multiple turns. Explanations aberrant from the correct interpretation and non-Chinese characters are highlighted with yellow bottom color. Escort Agency refers to historical security firms that were typically hired by trade merchants for protection during transportation of goods. They were known to use martial arts for defense, similar to bodyguards or security personnel.
• Figure 5: Answer distribution of models when answers are collected with free generation paradigm.