Brief but Impactful: How Human Tutoring Interactions Shape Engagement in Online Learning

TL;DR

This study investigates how brief, remote human tutoring visits influence engagement in a hybrid human–AI tutoring setting by minute-level alignment of tutor dialogue with MATHia system logs. Using a Poisson GLMM and a mixed-methods approach, it finds that visits elevate engagement during the visit (IRR = $1.61$) and yield modest post-visit uplift, with longer visits showing diminishing returns and later visits delivering larger immediate boosts. Qualitative and embedding-based analyses reveal that high-uplift dialogues feature concrete, stepwise scaffolding and explicit work organization, while low-uplift interactions rely on non-actionable praise. The findings offer practical guidance for resource-constrained classrooms—advocating an early, broad, and strategically timed set of brief tutor check-ins—and demonstrate a reproducible analytic pipeline linking dialogue content to moment-to-moment engagement dynamics.

Abstract

Learning analytics can guide human tutors to efficiently address motivational barriers to learning that AI systems struggle to support. Students become more engaged when they receive human attention. However, what occurs during short interventions, and when are they most effective? We align student-tutor dialogue transcripts with MATHia tutoring system log data to study brief human-tutor interactions on Zoom drawn from 2,075 hours of 191 middle school students' classroom math practice. Mixed-effect models reveal that engagement, measured as successful solution steps per minute, is higher during a human-tutor visit and remains elevated afterward. Visit length exhibits diminishing returns: engagement rises during and shortly after visits, irrespective of visit length. Timing also matters: later visits yield larger immediate lifts than earlier ones, though an early visit remains important to counteract engagement decline. We create analytics that identify which tutor-student dialogues raise engagement the most. Qualitative analysis reveals that interactions with concrete, stepwise scaffolding with explicit work organization elevate engagement most strongly. We discuss implications for resource-constrained tutoring, prioritizing several brief, well-timed check-ins by a human tutor while ensuring at least one early contact. Our analytics can guide the prioritization of students for support and surface effective tutor moves in real-time.

Figures (3)

• Figure 1: Visual representation of a student receiving remote tutor support while using MATHia. The Zoom screen capture (top half) is from a tutor visit, with the tutor and student (top right) grayed out to maintain anonymity. The bottom left shows the corresponding MATHia transaction logs, with the current log entry highlighted in blue. The bottom right presents the aligned WhisperX transcript, highlighting the tutor's instructional dialogue (also in blue) that corresponds to the student's actions.
• Figure 2: Descriptive plots validating key assumptions of our analysis. Left: Visits are associated with higher engagement after their event. Right: Students exhibited increased mastery with more learning opportunities (i.e., completed problem-solving steps), validating our choice of engagement measure.
• Figure 3: Engagement vs. tutoring episode duration. Blue points show empirical student-averaged steps/min for 5-minute duration bins; the orange line represents the estimated linear trend; the dotted line is the no-visit baseline. Labels indicate % increase over baseline at example durations 1, 5, 10, 15, and so forth.