Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Large Language Models' Ability Using a Psychiatric Screening Tool Based on Metaphor and Sarcasm Scenarios

Hiromu Yakura

TL;DR

This study evaluates large language models (LLMs) on figurative language using the Metaphor and Sarcasm Scenario Test (MSST), contrasting metaphor and sarcasm comprehension. By testing six publicly accessible LLMs (including GPT‑3.5, GPT‑4, Dolly 2.0, and several Llama 2 sizes) under identical prompts, the authors show that metaphor understanding improves with increasing model size, reaching levels comparable to younger humans, while sarcasm comprehension remains poor across models. The discussion ties these findings to developmental psychology, suggesting that metaphor relies on linguistic intelligence whereas sarcasm necessitates emotional intelligence and theory-of-mind, and it proposes data-efficient, multimodal, or emotion-focused training strategies to bridge this gap. Overall, the work highlights a clear gap in LLMs’ social-cognitive capabilities and offers avenues to guide future training that emphasizes emotional-context processing to enhance sarcasm understanding in AI systems.

Abstract

Metaphors and sarcasm are precious fruits of our highly evolved social communication skills. However, children with the condition then known as Asperger syndrome are known to have difficulties in comprehending sarcasm, even if they possess adequate verbal IQs for understanding metaphors. Accordingly, researchers had employed a screening test that assesses metaphor and sarcasm comprehension to distinguish Asperger syndrome from other conditions with similar external behaviors (e.g., attention-deficit/hyperactivity disorder). This study employs a standardized test to evaluate recent large language models' (LLMs) understanding of nuanced human communication. The results indicate improved metaphor comprehension with increased model parameters; however, no similar improvement was observed for sarcasm comprehension. Considering that a human's ability to grasp sarcasm has been associated with the amygdala, a pivotal cerebral region for emotional learning, a distinctive strategy for training LLMs would be imperative to imbue them with the ability in a cognitively grounded manner.

Evaluating Large Language Models' Ability Using a Psychiatric Screening Tool Based on Metaphor and Sarcasm Scenarios

TL;DR

This study evaluates large language models (LLMs) on figurative language using the Metaphor and Sarcasm Scenario Test (MSST), contrasting metaphor and sarcasm comprehension. By testing six publicly accessible LLMs (including GPT‑3.5, GPT‑4, Dolly 2.0, and several Llama 2 sizes) under identical prompts, the authors show that metaphor understanding improves with increasing model size, reaching levels comparable to younger humans, while sarcasm comprehension remains poor across models. The discussion ties these findings to developmental psychology, suggesting that metaphor relies on linguistic intelligence whereas sarcasm necessitates emotional intelligence and theory-of-mind, and it proposes data-efficient, multimodal, or emotion-focused training strategies to bridge this gap. Overall, the work highlights a clear gap in LLMs’ social-cognitive capabilities and offers avenues to guide future training that emphasizes emotional-context processing to enhance sarcasm understanding in AI systems.

Abstract

Metaphors and sarcasm are precious fruits of our highly evolved social communication skills. However, children with the condition then known as Asperger syndrome are known to have difficulties in comprehending sarcasm, even if they possess adequate verbal IQs for understanding metaphors. Accordingly, researchers had employed a screening test that assesses metaphor and sarcasm comprehension to distinguish Asperger syndrome from other conditions with similar external behaviors (e.g., attention-deficit/hyperactivity disorder). This study employs a standardized test to evaluate recent large language models' (LLMs) understanding of nuanced human communication. The results indicate improved metaphor comprehension with increased model parameters; however, no similar improvement was observed for sarcasm comprehension. Considering that a human's ability to grasp sarcasm has been associated with the amygdala, a pivotal cerebral region for emotional learning, a distinctive strategy for training LLMs would be imperative to imbue them with the ability in a cognitively grounded manner.
Paper Structure (10 sections, 3 figures, 1 table)

This paper contains 10 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Previous Research
  3. Research Questions
  4. Significance of The Topic
  5. Methods
  6. Material
  7. Procedure
  8. Results
  9. Discussion
  10. Conclusions

Figures (3)

  • Figure 1: The actual content of Q8 (taken from Adachi et al.Adachi2004). No LLM selected the right choice, i.e., (c).
  • Figure 2: The procedure of this study.
  • Figure 3: The comparison of the scores of LLMs on the MSST and the number of their parameters. Note that the parameter count of GPT-4 is not officially announced and is based on an online article Schreiner2023.