Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VideoAesBench: Benchmarking the Video Aesthetics Perception Capabilities of Large Multimodal Models

Yunhao Li, Sijing Wu, Zhilin Gao, Zicheng Zhang, Qi Jia, Huiyu Duan, Xiongkuo Min, Guangtao Zhai

TL;DR

VideoAesBench addresses the understudied problem of video aesthetics perception in large multimodal models by constructing a diverse, multi-dimensional benchmark. It collects 1,804 videos from UGC, AIGC, RGC, compression, and gaming sources and defines 12 fine-grained aesthetics dimensions across Visual Form, Visual Style, and Visual Affectiveness, with four question types designed to probe depth and explainability. The authors benchmark 23 open-source and commercial LMMs, revealing that current models offer only basic, uneven video aesthetics understanding, with closed-source models generally outperforming open-source ones and open-ended questions remaining particularly challenging. The benchmark provides a rigorous testbed for explainable video aesthetics assessment and offers actionable insights to guide future model development and evaluation in video perception tasks.

Abstract

Large multimodal models (LMMs) have demonstrated outstanding capabilities in various visual perception tasks, which has in turn made the evaluation of LMMs significant. However, the capability of video aesthetic quality assessment, which is a fundamental ability for human, remains underexplored for LMMs. To address this, we introduce VideoAesBench, a comprehensive benchmark for evaluating LMMs' understanding of video aesthetic quality. VideoAesBench has several significant characteristics: (1) Diverse content including 1,804 videos from multiple video sources including user-generated (UGC), AI-generated (AIGC), compressed, robotic-generated (RGC), and game videos. (2) Multiple question formats containing traditional single-choice questions, multi-choice questions, True or False questions, and a novel open-ended questions for video aesthetics description. (3) Holistic video aesthetics dimensions including visual form related questions from 5 aspects, visual style related questions from 4 aspects, and visual affectiveness questions from 3 aspects. Based on VideoAesBench, we benchmark 23 open-source and commercial large multimodal models. Our findings show that current LMMs only contain basic video aesthetics perception ability, their performance remains incomplete and imprecise. We hope our VideoAesBench can be served as a strong testbed and offer insights for explainable video aesthetics assessment.

VideoAesBench: Benchmarking the Video Aesthetics Perception Capabilities of Large Multimodal Models

TL;DR

VideoAesBench addresses the understudied problem of video aesthetics perception in large multimodal models by constructing a diverse, multi-dimensional benchmark. It collects 1,804 videos from UGC, AIGC, RGC, compression, and gaming sources and defines 12 fine-grained aesthetics dimensions across Visual Form, Visual Style, and Visual Affectiveness, with four question types designed to probe depth and explainability. The authors benchmark 23 open-source and commercial LMMs, revealing that current models offer only basic, uneven video aesthetics understanding, with closed-source models generally outperforming open-source ones and open-ended questions remaining particularly challenging. The benchmark provides a rigorous testbed for explainable video aesthetics assessment and offers actionable insights to guide future model development and evaluation in video perception tasks.

Abstract

Large multimodal models (LMMs) have demonstrated outstanding capabilities in various visual perception tasks, which has in turn made the evaluation of LMMs significant. However, the capability of video aesthetic quality assessment, which is a fundamental ability for human, remains underexplored for LMMs. To address this, we introduce VideoAesBench, a comprehensive benchmark for evaluating LMMs' understanding of video aesthetic quality. VideoAesBench has several significant characteristics: (1) Diverse content including 1,804 videos from multiple video sources including user-generated (UGC), AI-generated (AIGC), compressed, robotic-generated (RGC), and game videos. (2) Multiple question formats containing traditional single-choice questions, multi-choice questions, True or False questions, and a novel open-ended questions for video aesthetics description. (3) Holistic video aesthetics dimensions including visual form related questions from 5 aspects, visual style related questions from 4 aspects, and visual affectiveness questions from 3 aspects. Based on VideoAesBench, we benchmark 23 open-source and commercial large multimodal models. Our findings show that current LMMs only contain basic video aesthetics perception ability, their performance remains incomplete and imprecise. We hope our VideoAesBench can be served as a strong testbed and offer insights for explainable video aesthetics assessment.
Paper Structure (21 sections, 5 figures, 5 tables)

This paper contains 21 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarking Video LMMs
  4. Visual Aesthetics Assessment
  5. Visual Quality Assessment
  6. Benchmark Construction
  7. Aesthetic Video Collection
  8. Question Design
  9. Video Aesthetics Dimensions
  10. Visual Form.
  11. Visual Style.
  12. Visual Affectiveness.
  13. Question-Answer Pairs Construction
  14. VideoAesBench Experiments
  15. Experimental Setup
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Overview of our proposed VideoAesBench. The video are initially collected from five different scenarios including user-generated content (UGC) videos, AI-generated content (AIGC) videos, robot-generated content (RGC) videos, compression videos, and gaming videos. After obtaining the initial videos, we adopt a human-in-the-loop strategy to efficiently create high quality question-answer pairs for VideoAesBench. The concrete dataset distribution for question type, video content, and video aesthetics aspect are also depicted.
  • Figure 2: Data statistics of VideoAesBench. (a): Video resolution distribution across video width and height in VideoAesBench. (b): Distribution of video duration. (c): The word cloud statistics of all questions in VideoAesBench.
  • Figure 3: Visualization examples from VideoAesBench with in terms of aesthetics dimension. In our benchmark, the question type contains single-choice question, multiple-choice question, True-or-Flase question, and open-ended question. The aesthetic dimension concretely contains visual composition, visual elements and structure, shot size, depth of field, visual subject, lighting, color, visual tone, creativity, emotion, theme and communication, and viewer interest.
  • Figure 4: Performance comparison of MLLMs on VideoAesBench. (a) Overall performance across all evaluated MLLMs. (b) Performance of five representative MLLMs across different subclasses, including video sources, aesthetic dimensions, and question types.
  • Figure 5: GUI Figure.