EVQAScore: A Fine-grained Metric for Video Question Answering Data Quality Evaluation

TL;DR

EVQAScore addresses the need for high quality data in VideoLLMs by providing a reference-free evaluation metric for both video-caption and VideoQA data. It combines frame sampling, YOLO based key object extraction, and LLM driven keyword extraction to produce fine-grained precision-recall based scores that are efficient for long videos. The method achieves state-of-the-art correlations on VATEX-EVAL for caption data and demonstrates strong data filtering capabilities, enabling competitive VideoLLMs with substantially reduced data volumes. The results show that quality data selection with EVQAScore improves VideoLLM performance across multiple benchmarks and model architectures, highlighting practical impact for scalable video learning.

Abstract

Video question-answering (QA) is a core task in video understanding. Evaluating the quality of video QA and video caption data quality for training video large language models (VideoLLMs) is an essential challenge. Although various methods have been proposed for assessing video caption quality, there remains a lack of dedicated evaluation methods for Video QA. To address this gap, we introduce EVQAScore, a reference-free method that leverages keyword extraction to assess both video caption and video QA data quality. Additionally, we incorporate frame sampling and rescaling techniques to enhance the efficiency and robustness of our evaluation, this enables our score to evaluate the quality of extremely long videos. Our approach achieves state-of-the-art (SOTA) performance (32.8 for Kendall correlation and 42.3 for Spearman correlation, 4.7 and 5.9 higher than the previous method PAC-S++) on the VATEX-EVAL benchmark for video caption evaluation. Furthermore, by using EVQAScore for data selection, we achieved SOTA results with only 12.5\% of the original data volume, outperforming the previous SOTA method PAC-S and 100\% of data.

Figures (4)

• Figure 1: We provide an example of the EVQAScore evaluation process. In (a), we sample frames and apply YOLO for patch recognition. In (b), we extract keywords from the question-answer pairs. Finally, we calculate the EVQAScore based on the precision and recall between patches and keywords.
• Figure 2: We illustrate the computation of $\text{EVQAScore}_f$. First, fine-grained patches are extracted from the sampled frames. Next, keyword extraction is performed to enhance caption and question-answer semantic understanding. Finally, the patches and keywords are combined to calculate precision and recall, resulting in the EVQAScore.
• Figure 3: Prompt used for keywords extraction.
• Figure 4: In this case study, Answer1 is more accurate, while Answer2 introduces hallucination. Our keyword extraction method successfully identifies hallucinations, making EVQAScore the only metric capable of determining that Answer1 is superior to Answer2.