VideoLLM Benchmarks and Evaluation: A Survey

TL;DR

This survey examines benchmarks and evaluation methodologies for VideoLLMs, detailing closed-set, open-set, and specialized evaluations while tracing the evolution toward longer, multimodal, and domain-diverse benchmarks. It analyzes performance trends, architectural influences, and the gap between specialized VideoLLMs and general multimodal models, highlighting persistent challenges in long-form and temporal understanding. The authors propose future benchmark designs—hierarchical, multimodal, long-form narrative, interactive, robustness/adversarial, and explainability benchmarks—and discuss implications for ethics, efficiency, and interoperability across media, education, healthcare, and retail. The work emphasizes that robust, interpretable, and culturally aware evaluation frameworks are essential for deploying VideoLLMs responsibly and effectively in real-world applications.

Abstract

The rapid development of Large Language Models (LLMs) has catalyzed significant advancements in video understanding technologies. This survey provides a comprehensive analysis of benchmarks and evaluation methodologies specifically designed or used for Video Large Language Models (VideoLLMs). We examine the current landscape of video understanding benchmarks, discussing their characteristics, evaluation protocols, and limitations. The paper analyzes various evaluation methodologies, including closed-set, open-set, and specialized evaluations for temporal and spatiotemporal understanding tasks. We highlight the performance trends of state-of-the-art VideoLLMs across these benchmarks and identify key challenges in current evaluation frameworks. Additionally, we propose future research directions to enhance benchmark design, evaluation metrics, and protocols, including the need for more diverse, multimodal, and interpretability-focused benchmarks. This survey aims to equip researchers with a structured understanding of how to effectively evaluate VideoLLMs and identify promising avenues for advancing the field of video understanding with large language models.