Efficacy of a Computer Tutor that Models Expert Human Tutors
Andrew M. Olney, Sidney K. D'Mello, Natalie Person, Whitney Cade, Patrick Hays, Claire W. Dempsey, Blair Lehman, Betsy Williams, Art Graesser
TL;DR
This study evaluates whether an intelligent tutoring system (Guru) modeled on expert human tutors provides learning gains comparable to expert tutors and to classroom instruction in high school biology. Using a 9‑week, three-condition repeated-measures design and logistic mixed-effects analysis, the results show that both Guru and human tutors outperform the classroom control on immediate and delayed assessments, with no significant difference between Guru and human tutors. The large immediate effects for both tutoring conditions and the robustness of results across analyses support the notion that expert tutoring strategies can be effectively emulated in an ITS, though distinguishing domain expertise from tutoring expertise remains a challenge. The findings have implications for designing future ITSs and for assessing tutoring effectiveness in real-world educational settings, including the potential for AI-generated dynamic assessments to separate expertise components.
Abstract
Tutoring is highly effective for promoting learning. However, the contribution of expertise to tutoring effectiveness is unclear and continues to be debated. We conducted a 9-week learning efficacy study of an intelligent tutoring system (ITS) for biology modeled on expert human tutors with two control conditions: human tutors who were experts in the domain but not in tutoring and a no-tutoring condition. All conditions were supplemental to classroom instruction, and students took learning tests immediately before and after tutoring sessions as well as delayed tests 1-2 weeks later. Analysis using logistic mixed-effects modeling indicates significant positive effects on the immediate post-test for the ITS (d =.71) and human tutors (d =.66) which are in the 99th percentile of meta-analytic effects, as well as significant positive effects on the delayed post-test for the ITS (d =.36) and human tutors (d =.39). We discuss implications for the role of expertise in tutoring and the design of future studies.