Oogiri-Master: Benchmarking Humor Understanding via Oogiri

TL;DR

We address how to evaluate humor understanding in large language models by leveraging the Japanese Oogiri game. We introduce Oogiri-Corpus, a large, bias-controlled dataset with ~100 candidate responses per prompt and independent funniness ratings from ~100 judges, enabling objective analysis of humor components. Through quantitative linguistic-feature analysis (including length, ambiguity, incongruity resolution, surprisal, and NLI signals) we identify predictors of funniness and build an Oogiri-Master benchmark with five tasks (four relative MCQA and one absolute classification). Benchmark results show state-of-the-art LLMs approaching human performance, with insight-augmented prompting and continued Japanese pretraining further boosting performance; instruction design also shapes outcomes. These findings provide a principled framework for evaluating and advancing humor understanding in multilingual LLMs and guide future work on cross-lingual and multimodal humor tasks.

Abstract

Humor is a salient testbed for human-like creative thinking in large language models (LLMs). We study humor using the Japanese creative response game Oogiri, in which participants produce witty responses to a given prompt, and ask the following research question: What makes such responses funny to humans? Previous work has offered only limited reliable means to answer this question. Existing datasets contain few candidate responses per prompt, expose popularity signals during ratings, and lack objective and comparable metrics for funniness. Thus, we introduce Oogiri-Master and Oogiri-Corpus, which are a benchmark and dataset designed to enable rigorous evaluation of humor understanding in LLMs. Each prompt is paired with approximately 100 diverse candidate responses, and funniness is rated independently by approximately 100 human judges without access to others' ratings, reducing popularity bias and enabling robust aggregation. Using Oogiri-Corpus, we conduct a quantitative analysis of the linguistic factors associated with funniness, such as text length, ambiguity, and incongruity resolution, and derive objective metrics for predicting human judgments. Subsequently, we benchmark a range of LLMs and human baselines in Oogiri-Master, demonstrating that state-of-the-art models approach human performance and that insight-augmented prompting improves the model performance. Our results provide a principled basis for evaluating and advancing humor understanding in LLMs.

Figures (3)

• Figure 1: Oogiri prompt--response example.
• Figure 2: Prompt for the absolute judgment task.
• Figure 3: Prompt for LLM-based scoring of higher-order linguistic features.