Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking

Xilin Jiang, Qiaolin Wang, Junkai Wu, Xiaomin He, Zhongweiyang Xu, Yinghao Ma, Minshuo Piao, Kaiyi Yang, Xiuwen Zheng, Riki Shimizu, Yicong Chen, Arsalan Firoozi, Gavin Mischler, Sukru Samet Dindar, Richard Antonello, Linyang He, Tsun-An Hsieh, Xulin Fan, Yulun Wu, Yuesheng Ma, Chaitanya Amballa, Weixiong Chen, Jiarui Hai, Ruisi Li, Vishal Choudhari, Cong Han, Yinghao Aaron Li, Adeen Flinker, Mounya Elhilali, Emmanouil Benetos, Mark Hasegawa-Johnson, Romit Roy Choudhury, Nima Mesgarani

TL;DR

AVMeme Exam introduces a large-scale, multimodal, multilingual benchmark of 1,032 audio-visual memes to probe not only surface content but also context, emotion, usage, and world knowledge. The dataset employs a rigorous collection and verification pipeline with seven question types designed to stress-contextual and cultural understanding, and includes thorough cheat-detection to ensure authentic multimodal reasoning. Across 19 state-of-the-art MLLMs, results reveal strong surface-level performance on linguistic content but substantial gaps in textless audio interpretation, contextual inference, and cultural grounding, with even top models lagging behind human performance on many tasks. The work highlights a key gap in current multimodal AI and provides a path toward more human-aligned, culturally aware multimodal intelligence through richer annotations and evaluation protocols.

Abstract

Internet audio-visual clips convey meaning through time-varying sound and motion, which extend beyond what text alone can represent. To examine whether AI models can understand such signals in human cultural contexts, we introduce AVMeme Exam, a human-curated benchmark of over one thousand iconic Internet sounds and videos spanning speech, songs, music, and sound effects. Each meme is paired with a unique Q&A assessing levels of understanding from surface content to context and emotion to usage and world knowledge, along with metadata such as original year, transcript, summary, and sensitivity. We systematically evaluate state-of-the-art multimodal large language models (MLLMs) alongside human participants using this benchmark. Our results reveal a consistent limitation: current models perform poorly on textless music and sound effects, and struggle to think in context and in culture compared to surface content. These findings highlight a key gap in human-aligned multimodal intelligence and call for models that can perceive contextually and culturally beyond the surface of what they hear and see. Project page: avmemeexam.github.io/public

AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking

TL;DR

AVMeme Exam introduces a large-scale, multimodal, multilingual benchmark of 1,032 audio-visual memes to probe not only surface content but also context, emotion, usage, and world knowledge. The dataset employs a rigorous collection and verification pipeline with seven question types designed to stress-contextual and cultural understanding, and includes thorough cheat-detection to ensure authentic multimodal reasoning. Across 19 state-of-the-art MLLMs, results reveal strong surface-level performance on linguistic content but substantial gaps in textless audio interpretation, contextual inference, and cultural grounding, with even top models lagging behind human performance on many tasks. The work highlights a key gap in current multimodal AI and provides a path toward more human-aligned, culturally aware multimodal intelligence through richer annotations and evaluation protocols.

Abstract

Internet audio-visual clips convey meaning through time-varying sound and motion, which extend beyond what text alone can represent. To examine whether AI models can understand such signals in human cultural contexts, we introduce AVMeme Exam, a human-curated benchmark of over one thousand iconic Internet sounds and videos spanning speech, songs, music, and sound effects. Each meme is paired with a unique Q&A assessing levels of understanding from surface content to context and emotion to usage and world knowledge, along with metadata such as original year, transcript, summary, and sensitivity. We systematically evaluate state-of-the-art multimodal large language models (MLLMs) alongside human participants using this benchmark. Our results reveal a consistent limitation: current models perform poorly on textless music and sound effects, and struggle to think in context and in culture compared to surface content. These findings highlight a key gap in human-aligned multimodal intelligence and call for models that can perceive contextually and culturally beyond the surface of what they hear and see. Project page: avmemeexam.github.io/public
Paper Structure (14 sections, 8 figures, 5 tables)

This paper contains 14 sections, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. AVMeme Exam
  3. Collection
  4. Verification
  5. Question Types
  6. Main Results
  7. Conclusion
  8. Collection & Verification Procedures
  9. Human Evaluation Details
  10. Supplementary Results
  11. Study on On-screen Text
  12. Study on Meme's Year
  13. Correct and Wrong Thinking Example
  14. More AVMeme Samples

Figures (8)

  • Figure 1: AVMeme Exam includes seven question types covering content, context, and world knowledge of audio-visual signals. We find that while multimodal large langugage models perform strongest on surface linguistic tasks, they struggle with contextual inference, world knowledge, and interpreting textless audio.
  • Figure 2: Top: Historical timeline of the 1,032 audio-visual memes curated in AVMeme Exam, spotlighting famous music rhythm, movie lines, sound effects, and viral Internet memes. Bottom: Pie charts summarize the distributions of question types, sound categories, and languages, highlighting the data diversity. Right: Frequent words in the memes' names and distributions of clip durations and number of choices. The duration is cut to 30 seconds, which is the maximum input audio length for most models.
  • Figure 3: AVMeme Exam collection & verification pipeline. Videos and Q&As are human collected and verified (yellow). LLMs (gray) are used for text cleanup and to detect questions easily answered by text without audio given.
  • Figure 4: Models vs. human individuals, grouped by humans are faimilar/unfamilar/unseen the meme.
  • Figure 5: A mosaic art of "AVMeme Exam" dotted by image frames from the meme clips.
  • ...and 3 more figures