Pragmatic Metacognitive Prompting Improves LLM Performance on Sarcasm Detection

TL;DR

Sarcasm detection in sentiment analysis is difficult due to nonliteral language. The authors introduce Pragmatic Metacognitive Prompting (PMP), combining pragmatic theories with metacognitive prompting to guide LLMs through reasoning about implied meaning and context, followed by reflection to produce a final judgment. Across benchmarks MUStARD and SemEval-2018 Task 3, PMP achieves state-of-the-art performance with GPT-4o and shows competitive results with other models, outperforming several standard prompting methods. The study demonstrates that integrating linguistic pragmatics and reflective reasoning into prompting can substantially enhance sarcasm detection, suggesting a fruitful direction for future sentiment-analysis research.

Abstract

Sarcasm detection is a significant challenge in sentiment analysis due to the nuanced and context-dependent nature of verbiage. We introduce Pragmatic Metacognitive Prompting (PMP) to improve the performance of Large Language Models (LLMs) in sarcasm detection, which leverages principles from pragmatics and reflection helping LLMs interpret implied meanings, consider contextual cues, and reflect on discrepancies to identify sarcasm. Using state-of-the-art LLMs such as LLaMA-3-8B, GPT-4o, and Claude 3.5 Sonnet, PMP achieves state-of-the-art performance on GPT-4o on MUStARD and SemEval2018. This study demonstrates that integrating pragmatic reasoning and metacognitive strategies into prompting significantly enhances LLMs' ability to detect sarcasm, offering a promising direction for future research in sentiment analysis.

Figures (1)

• Figure 2: PMP analysis on the phrase "Lots of people tweeting pictures from their cars of their snowy commutes to work, whilst saying 'stay safe' Oh, the #irony!".