LecEval: An Automated Metric for Multimodal Knowledge Acquisition in Multimedia Learning

TL;DR

LecEval addresses the challenge of evaluating slide-based multimodal instruction by introducing a theory-grounded automated metric anchored in Mayer's principles of multimedia learning. It defines four rubrics—$CR$, $EC$, $LS$, and $AE$—and builds a large-scale, fine-grained dataset of over $2{,}000$ slide–text samples from more than $50$ online courses, paired with human annotations. A fine-tuned multimodal backbone trained on this data yields correlations with human judgments that surpass both reference-based and prompt-based baselines, approaching inter-annotator agreement. The work provides a practical, scalable tool and releases the dataset, processing toolkit, and a trained reward model to enable reproducibility and broader impact in evaluating multimodal educational content.

Abstract

Evaluating the quality of slide-based multimedia instruction is challenging. Existing methods like manual assessment, reference-based metrics, and large language model evaluators face limitations in scalability, context capture, or bias. In this paper, we introduce LecEval, an automated metric grounded in Mayer's Cognitive Theory of Multimedia Learning, to evaluate multimodal knowledge acquisition in slide-based learning. LecEval assesses effectiveness using four rubrics: Content Relevance (CR), Expressive Clarity (EC), Logical Structure (LS), and Audience Engagement (AE). We curate a large-scale dataset of over 2,000 slides from more than 50 online course videos, annotated with fine-grained human ratings across these rubrics. A model trained on this dataset demonstrates superior accuracy and adaptability compared to existing metrics, bridging the gap between automated and human assessments. We release our dataset and toolkits at https://github.com/JoylimJY/LecEval.

Figures (3)

• Figure 1: The general data construction framework of our dataset. (1) Heterogeneous Data Integration: We collect and process online lecture videos, extracting both slides and speech transcripts. (2) Multimodal Alignment and Refinement. We manually align transcriptions with their corresponding slides and refine the raw transcriptions leveraging GPT-4. (3) Fine-grained Assessment. We engage experienced human annotators to evaluate the slide presentations across our predefined rubrics.
• Figure 2: Spearman correlations ($\rho$) between training data scalability and model's performance.
• Figure 3: Correlations between (1) scoring function, and (2) criterion-specific modeling with model's performance.