Revolutionising Distance Learning: A Comparative Study of Learning Progress with AI-Driven Tutoring

TL;DR

First evidence that generative AI can increase the speed of learning substantially in university students is presented, and the magnitude of the effect and the scalability of the approach implicate generative AI as a key lever to significantly improve and accelerate learning by personalisation.

Abstract

Generative AI is expected to have a vast, positive impact on education; however, at present, this potential has not yet been demonstrated at scale at university level. In this study, we present first evidence that generative AI can increase the speed of learning substantially in university students. We tested whether using the AI-powered teaching assistant Syntea affected the speed of learning of hundreds of distance learning students across more than 40 courses at the IU International University of Applied Sciences. Our analysis suggests that using Syntea reduced their study time substantially--by about 27\% on average--in the third month after the release of Syntea. Taken together, the magnitude of the effect and the scalability of the approach implicate generative AI as a key lever to significantly improve and accelerate learning by personalisation.

Figures (3)

• Figure 1: Explorative analysis. A) 60-day moving window average of study progression (y-axis) over time (x-axis) for the treatment group (orange) and the control group (blue). We also show the difference between the two groups (green). The release is marked by a dashed vertical line. B) Relative difference between treatment and control group (y-axis) over time (x-axis), with the release date marked by a vertical dashed line.
• Figure 2: Explorative analysis without new joiners. We show the exact same data as in Figure \ref{['fig:progress']} above, with one exception: we excluded all students that have started in or after January 2023, from both the control group and the treatment group. We find that the main dynamical features persist (compare Figure 1). In particular, we find that the widening gap between control and treatment group that follows the release date is present, and in fact even more pronounced with new joiners excluded.
• Figure 3: Average study duration before and after Syntea rollout. We show a summary of our main results, taken from Table \ref{['tab:progression']}. We focus on the average study duration (using units of months/exam) of both the control group and the treatment group (called 'Syntea group' above), comparing the period before the release of Syntea (the months -24 to -1 prior to release, 10/21 - 09/23) with the most recent period (month 2 post release, 12/23). The expected value for the treatment group after the rollout is based on the ratio between treatment group and control group in the pre-rollout period (80.6%), which is extrapolated to the post-rollout period (yielding a prediction of $2.77 \textrm{ months/exam}\times 80.6\% = 2.23 \textrm{ months/exam}$). The relative difference between this expected value and the measured value of 1.63 months/exam constitutes a reduction in average study time of 27%.