V-HUB: A Visual-Centric Humor Understanding Benchmark for Video LLMs

TL;DR

v-HUB introduces a visual-centric humor benchmark for video LLMs by curating humor-centric clips from Charlie Chaplin and user-generated videos, all emphasizing purely visual humor. It defines three tasks—Caption Matching, Humor Explanation, and Open-ended QA—and evaluates a spectrum of MLLMs under Text-Only, Video-Only, and Video+Audio settings, illustrating strong reliance on linguistic cues and the benefits of audio and background knowledge for humor understanding. The results reveal substantial gaps in visual reasoning for humor: models struggle with humor discovery, exhibit limited cross-modal fusion, and falter on subtle humor; audio and textual cues can partially mitigate these gaps. The benchmark provides a valuable platform for diagnosing and driving progress in cross-modal humor understanding, with practical implications for human-AI interaction, content understanding, and cross-cultural communication.

Abstract

AI models capable of comprehending humor hold real-world promise -- for example, enhancing engagement in human-machine interactions. To gauge and diagnose the capacity of multimodal large language models (MLLMs) for humor understanding, we introduce v-HUB, a novel visual-centric video humor understanding benchmark. v-HUB comprises a curated collection of minimally verbal short videos, sourced from classic silent films and online resources, and reflecting real-world scenarios where humor can be appreciated purely through visual cues. Each video clip is paired with rich annotations, including captions, descriptions, and explanations, supporting evaluation tasks like caption matching and humor explanation. To broaden its applicability, we further construct an open-ended video QA task, making it readily integrable into existing video understanding benchmarks. We evaluate a diverse set of MLLMs, from specialized Video-LLMs to versatile OmniLLMs that can process audio, covering both open-source and proprietary domains. The experimental results expose the difficulties MLLMs face in comprehending humor from visual cues alone. For example, all models exhibit a marked performance drop on caption matching when moving from text-based to video-based evaluation (without audio). Our findings also demonstrate that incorporating audio helps with video humor understanding, highlighting the informativeness of sound and the promise of integrating richer modalities for complex video understanding tasks.

Figures (8)

• Figure 1: Examples of visual-centric humor understanding, where 'audio' and 'text' refer to environmental sound (cf. human speech) and visual text, respectively.
• Figure 2: Data Curation Pipeline. To collect visual-centric humorous videos, the pipeline consists of two main stages: (a) Humorous video collection, where annotators identify timestamps of self-contained humorous clips for silent films and verify humor presence in short videos (see Section \ref{['sec:data-collection']}). (b) Filtering and annotation, where only visual-dominant humor is retained and annotated (see Section \ref{['sec:data-annotation']}). The annotation is further used for task construction (see Section \ref{['sec:tasks']}).
• Figure 3: Example annotation of a short video that conveys humor through visuals, visual text, and background sound. Knowing the Happy Birthday melody makes the video merrier (see Section \ref{['sec:data-annotation']}).
• Figure 4: Interface for the Harmful Content Detection HIT.
• Figure 5: Interface for HIT.