Assessing the Capabilities of LLMs in Humor:A Multi-dimensional Analysis of Oogiri Generation and Evaluation

TL;DR

This paper argues that humor in AI cannot be captured by a single metric and introduces a multidimensional framework rooted in Japanese Oogiri to study LLMs’ humor generation and evaluation. By expanding Oogiri-GO and Oogiri-Chaya and annotating responses across six dimensions, the authors show that current LLMs can generate humor at roughly low-to-mid human levels but notably lag in Empathy, which underpins effective humorous resolution. They reveal a fundamental divergence in evaluation criteria between humans and LLMs: humans prize Empathy while LLMs emphasize Novelty, leading to limited overall agreement and model-specific biases. The study provides a new, publicly released annotated corpus and insights that can guide the development of more emotionally intelligent conversational agents and more robust humor evaluation methods.

Abstract

Computational humor is a frontier for creating advanced and engaging natural language processing (NLP) applications, such as sophisticated dialogue systems. While previous studies have benchmarked the humor capabilities of Large Language Models (LLMs), they have often relied on single-dimensional evaluations, such as judging whether something is simply ``funny.'' This paper argues that a multifaceted understanding of humor is necessary and addresses this gap by systematically evaluating LLMs through the lens of Oogiri, a form of Japanese improvisational comedy games. To achieve this, we expanded upon existing Oogiri datasets with data from new sources and then augmented the collection with Oogiri responses generated by LLMs. We then manually annotated this expanded collection with 5-point absolute ratings across six dimensions: Novelty, Clarity, Relevance, Intelligence, Empathy, and Overall Funniness. Using this dataset, we assessed the capabilities of state-of-the-art LLMs on two core tasks: their ability to generate creative Oogiri responses and their ability to evaluate the funniness of responses using a six-dimensional evaluation. Our results show that while LLMs can generate responses at a level between low- and mid-tier human performance, they exhibit a notable lack of Empathy. This deficit in Empathy helps explain their failure to replicate human humor assessment. Correlation analyses of human and model evaluation data further reveal a fundamental divergence in evaluation criteria: LLMs prioritize Novelty, whereas humans prioritize Empathy. We release our annotated corpus to the community to pave the way for the development of more emotionally intelligent and sophisticated conversational agents.

Figures (14)

• Figure 1: Multi-dimensional comparison of Oogiri responses from humans and LLMs, as evaluated by humans. "Human high tier" and "human low tier" refer to the highest- and lowest-rated human responses, as determined by manual annotation. (Funniness refers to Overall Funniness.)
• Figure 2: Overview of the experimental process for Oogiri generation and evaluation.
• Figure 3: Spearman correlation matrix of human and GPT-4.1 evaluation. Human evaluations are in the lower triangle, and GPT-4.1 evaluations are in the upper triangle. GPT-4.1 is a representative example; see Appendix E for other models.
• Figure 4: Human vs. LLM ratings: Scatter plot of Overall Funniness. Solid colored lines show the linear regression trend for each response category, with shaded areas indicating the 95 % confidence interval. The dashed line ($y=x$) represents perfect agreement. GPT-4.1 is a representative example; see Appendix D for results from all models.
• Figure 5: Prompt template of Oogiri generation and evaluation. The original prompt was in Japanese and has been translated here for clarity.