Learning Outcomes, Assessment, and Evaluation in Educational Recommender Systems: A Systematic Review
Nursultan Askarbekuly, Ivan Luković
TL;DR
This systematic literature review investigates how Educational Recommender Systems (ERS) measure and optimize learning, emphasizing target metrics, evaluation methods, and outcome-based assessment. Analyzing 28 papers selected from 1,395 candidates, the study finds a predominance of rating-based metrics and a minority of learning-centered evaluations, with outcome-based assessment applied in roughly a third of cases and mainly within formal university courses. The authors argue that there is a significant gap in assessing the pedagogical impact of ERS at scale and in informal education contexts, highlighting the need for comprehensive evaluation criteria that align learning outcomes with assessment and system performance. They propose focusing on learning-oriented metrics, developing scalable outcome-based assessment methods, and improving educational datasets to enable more direct measurement of learning gains and competencies.
Abstract
In this paper, we analyse how learning is measured and optimized in Educational Recommender Systems (ERS). In particular, we examine the target metrics and evaluation methods used in the existing ERS research, with a particular focus on the pedagogical effect of recommendations. While conducting this systematic literature review (SLR), we identified 1395 potentially relevant papers, then filtered them through the inclusion and exclusion criteria, and finally selected and analyzed 28 relevant papers. Rating-based relevance is the most popular target metric, while less than a half of papers optimize learning-based metrics. Only a third of the papers used outcome-based assessment to measure the pedagogical effect of recommendations, mostly within a formal university course. This indicates a gap in ERS research with respect to assessing the pedagogical effect of recommendations at scale and in informal education settings.