AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms

TL;DR

The study investigates whether a pedagogy-tuned generative AI tutor, LearnLM, can safely and effectively scale one-to-one tutoring in UK classrooms. Using a two-level randomized classroom trial within the Eedi platform, the authors demonstrate that AI-assisted tutoring supervised by human experts yields learning outcomes comparable to or better than human tutoring alone, and in particular enhances knowledge transfer to novel topics by about 5.5 percentage points. A rigorous safety audit found LearnLM drafts were typically adopted with minimal edits and produced virtually no harmful content. The findings suggest that pedagogically constrained AI tutoring can complement teachers, improve scalability, and support durable, transferable learning, while highlighting the need for longitudinal, cross-domain research and careful attention to pacing and social-emotional nuance. This work advances evidence on how AI tutors, when properly supervised, can deliver effective, scalable, and safe personalized instruction in real classrooms.

Abstract

One-to-one tutoring is widely considered the gold standard for personalized education, yet it remains prohibitively expensive to scale. To evaluate whether generative AI might help expand access to this resource, we conducted an exploratory randomized controlled trial (RCT) with $N = 165$ students across five UK secondary schools. We integrated LearnLM -- a generative AI model fine-tuned for pedagogy -- into chat-based tutoring sessions on the Eedi mathematics platform. In the RCT, expert tutors directly supervised LearnLM, with the remit to revise each message it drafted until they would be satisfied sending it themselves. LearnLM proved to be a reliable source of pedagogical instruction, with supervising tutors approving 76.4% of its drafted messages making zero or minimal edits (i.e., changing only one or two characters). This translated into effective tutoring support: students guided by LearnLM performed at least as well as students chatting with human tutors on each learning outcome we measured. In fact, students who received support from LearnLM were 5.5 percentage points more likely to solve novel problems on subsequent topics (with a success rate of 66.2%) than those who received tutoring from human tutors alone (rate of 60.7%). In interviews, tutors highlighted LearnLM's strength at drafting Socratic questions that encouraged deeper reflection from students, with multiple tutors even reporting that they learned new pedagogical practices from the model. Overall, our results suggest that pedagogically fine-tuned AI tutoring systems may play a promising role in delivering effective, individualized learning support at scale.

Figures (5)

• Figure 1: We designed this exploratory RCT to evaluate the safety, pedagogy, and efficacy of LearnLM. (a) The RCT randomly assigned each of $N = 165$ students to receive either static hints or interactive tutoring. Students in the tutoring condition experienced a further level of randomization. When they started a tutoring session, the platform randomly assigned them to either a session with a human tutor or a session with LearnLM (supervised by a human tutor). This design allows us to compare static, pre-written support against interactive tutoring, as well as human tutoring against (supervised) tutoring from LearnLM. (b) In sessions with LearnLM, a supervising tutor reviewed each message that LearnLM drafted. They could either edit the message, completely re-write it, or approve it without any changes. The Eedi platform then sent the message to the student.
• Figure 2: Student progression through the study unit. If a student makes a mistake on the first question in a study unit, they receive a support intervention. We analyze whether the intervention helps the student identify and remediate their mistake, resolve the misconception underlying their incorrect choice, and transfer the knowledge from the intervention to the next study unit. See \ref{['sec:methods']} and Appendix \ref{['sec:appendix/platform']} for more information on the Eedi platform.
• Figure 3: Tutoring interventions improve student learning outcomes. (left, center) For immediate learning outcomes, sessions with human tutors and expert-supervised sessions with LearnLM promote similar growth for students. Students who receive interactive tutoring from either source substantially outperform students who receive pre-written, static hints. (right) In contrast, students tutored by LearnLM demonstrate greater knowledge transfer to new topics than those supported either by static hints or by human tutors alone. Error bars indicate 95% credible intervals. Dashed lines represent the chance of success when guessing randomly (33.3%, 66.7%, and 25%, respectively).
• Figure B.1: Our RCT focused on two support interventions on the Eedi platform. After making a mistake in a study unit, students in the control condition received static hints (left), which deliver immediate, pre-written feedback targeting the specific misconception underlying the incorrect answer they chose. Students in the tutoring condition (right) received one-to-one, chat-based assistance from a tutor.
• Figure I.1: Transcript of an example supervised tutoring session with LearnLM. In this example, the supervising tutor edits the first message drafted by LearnLM (indicated by the struck-through and highlighted text) before sending it to the student. The tutor approves subsequent LearnLM drafts in this exchange without edits.