Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GODBench: A Benchmark for Multimodal Large Language Models in Video Comment Art

Yiming Lei, Chenkai Zhang, Zeming Liu, Haitao Leng, Shaoguo Liu, Tingting Gao, Qingjie Liu, Yunhong Wang

TL;DR

GODBench addresses the lack of comprehensive evaluation for creative Video Comment Art by providing a large-scale, diverse multimodal dataset of 67,073 videos with GOD-level comments across 31 categories, plus five dimensions of Comment Art and two task types. The authors introduce Ripple of Thought (RoT), a five-phase reasoning framework that expands the knowledge space through Ripple Initiation, Focalization, Diffusion, Wave Interference, and Luminous Imprint to improve creativity in generation and analysis. Extensive experiments across 10 open-/commercial Video-LLMs show RoT substantially boosts discriminative and generative performance, even surpassing baselines on automated metrics and aligning with human judgments in many cases. The work also provides a rigorous annotation protocol, evaluation methodology, and analysis tools (including the Weighted Entity Overlap) to quantify divergent associations and creative quality, signaling a path toward more creative and culturally aware multimodal systems.

Abstract

Video Comment Art enhances user engagement by providing creative content that conveys humor, satire, or emotional resonance, requiring a nuanced and comprehensive grasp of cultural and contextual subtleties. Although Multimodal Large Language Models (MLLMs) and Chain-of-Thought (CoT) have demonstrated strong reasoning abilities in STEM tasks (e.g. mathematics and coding), they still struggle to generate creative expressions such as resonant jokes and insightful satire. Moreover, existing benchmarks are constrained by their limited modalities and insufficient categories, hindering the exploration of comprehensive creativity in video-based Comment Art creation. To address these limitations, we introduce GODBench, a novel benchmark that integrates video and text modalities to systematically evaluate MLLMs' abilities to compose Comment Art. Furthermore, inspired by the propagation patterns of waves in physics, we propose Ripple of Thought (RoT), a multi-step reasoning framework designed to enhance the creativity of MLLMs. Extensive experiments reveal that existing MLLMs and CoT methods still face significant challenges in understanding and generating creative video comments. In contrast, RoT provides an effective approach to improve creative composing, highlighting its potential to drive meaningful advancements in MLLM-based creativity. GODBench is publicly available at https://github.com/stan-lei/GODBench-ACL2025.

GODBench: A Benchmark for Multimodal Large Language Models in Video Comment Art

TL;DR

GODBench addresses the lack of comprehensive evaluation for creative Video Comment Art by providing a large-scale, diverse multimodal dataset of 67,073 videos with GOD-level comments across 31 categories, plus five dimensions of Comment Art and two task types. The authors introduce Ripple of Thought (RoT), a five-phase reasoning framework that expands the knowledge space through Ripple Initiation, Focalization, Diffusion, Wave Interference, and Luminous Imprint to improve creativity in generation and analysis. Extensive experiments across 10 open-/commercial Video-LLMs show RoT substantially boosts discriminative and generative performance, even surpassing baselines on automated metrics and aligning with human judgments in many cases. The work also provides a rigorous annotation protocol, evaluation methodology, and analysis tools (including the Weighted Entity Overlap) to quantify divergent associations and creative quality, signaling a path toward more creative and culturally aware multimodal systems.

Abstract

Video Comment Art enhances user engagement by providing creative content that conveys humor, satire, or emotional resonance, requiring a nuanced and comprehensive grasp of cultural and contextual subtleties. Although Multimodal Large Language Models (MLLMs) and Chain-of-Thought (CoT) have demonstrated strong reasoning abilities in STEM tasks (e.g. mathematics and coding), they still struggle to generate creative expressions such as resonant jokes and insightful satire. Moreover, existing benchmarks are constrained by their limited modalities and insufficient categories, hindering the exploration of comprehensive creativity in video-based Comment Art creation. To address these limitations, we introduce GODBench, a novel benchmark that integrates video and text modalities to systematically evaluate MLLMs' abilities to compose Comment Art. Furthermore, inspired by the propagation patterns of waves in physics, we propose Ripple of Thought (RoT), a multi-step reasoning framework designed to enhance the creativity of MLLMs. Extensive experiments reveal that existing MLLMs and CoT methods still face significant challenges in understanding and generating creative video comments. In contrast, RoT provides an effective approach to improve creative composing, highlighting its potential to drive meaningful advancements in MLLM-based creativity. GODBench is publicly available at https://github.com/stan-lei/GODBench-ACL2025.
Paper Structure (41 sections, 16 equations, 66 figures, 11 tables)

This paper contains 41 sections, 16 equations, 66 figures, 11 tables.

Table of Contents

  1. Introduction
  2. GODBench
  3. Dataset Construction
  4. Comment Art Definition and Annotation
  5. Evaluation Procedure and Metrics
  6. Discriminative Tasks
  7. Generative Tasks
  8. Ripple of Thought
  9. Ripple Initiation
  10. Ripple Focalization
  11. Ripple Diffusion
  12. Wave Interference
  13. Luminous Imprint
  14. Experiment
  15. Experimental Setups
  16. ...and 26 more sections

Figures (66)

  • Figure 1: Example from GODBench. Showcasing a human-written GOD-level comment for the video, alongside the comments generated by model using the RoT framework and standard model. "#" indicates that the original text is in Chinese.
  • Figure 2: The detailed definition of Comment Art and example of various tasks.Comment Art is defined in five dimensions, each with different subcategories. One specific example of "Imaginary Completion" is presented, including the input video and various discriminative and generative tasks.
  • Figure 3: Illustration of RoT. Human creative thinking is like the diffusion of ripples, breaking down the propagation of waves in physics into five components, which are then transferred to the RoT reasoning framework of MLLMs.
  • Figure 4: User study with voting(%) by different models and improved methods.
  • Figure 5: Procedure and results of WEO score on divergent association tasks in GODBench.
  • ...and 61 more figures