Stabilising Learner Trajectories: A Doubly Robust Evaluation of AI-Guided Student Support using Activity Theory

TL;DR

The paper evaluates an AI-guided proactive student-support system using doubly robust propensity score matching within an Activity Theory framework. By time-aligning AI-derived probability of success scores and matching treated at-risk students to similar controls, the study mitigates immortal time bias and reveals causal associations with academic outcomes. Results show reduced course failures and higher cumulative grades for supported students, though effects on qualification pace are less certain due to institutional constraints. Activity Theory helps interpret these findings as a socio-technical brake that stabilises at-risk trajectories, with recommendations to align governance and curriculum structures to convert stability into faster degree progression. The work advances learning analytics evaluation by integrating rigorous causal methods with a socio-technical lens, offering practical guidance for designing governance and intervention workflows in higher education.

Abstract

While predictive models are increasingly common in higher education, causal evidence regarding the interventions they trigger remains rare. This study evaluates an AI-guided student support system at a large university using doubly robust propensity score matching. We advance the methodology for learning analytics evaluation by leveraging time-aligned, dynamic AI probability of success scores to match 1,859 treated students to controls, thereby mitigating the selection and immortal time biases often overlooked in observational studies. Results indicate that the intervention effectively stabilised precarious trajectories, and compared to the control group, supported students significantly reduced their course failure rates and achieved higher cumulative grades. However, effects on the speed of qualification completion were positive but statistically constrained. We interpreted these findings through Activity Theory, framing the intervention as a socio-technical brake that interrupts and slows the accumulation of academic failure among at-risk students. The student support-AI configuration successfully resolved the primary contradiction of immediate academic risk, but secondary contradictions within institutional structures limited the acceleration of degree completion. We conclude that while AI-enabled support effectively arrests decline, translating this stability into faster progression requires aligning intervention strategies with broader institutional governance.

Figures (11)

• Figure 1: The Activity Theory framework(left) applied to the study's learning analytics context (right).
• Figure 2: SHAP summary plot for the PoS model. Each point is a student record; position on the horizontal axis indicates the impact of the feature on predicted success, and colour indicates the feature value (blue = low, red = high). Features are ordered by overall importance.
• Figure 3: Density of AI probability of success (PoS) for treated and matched control students after propensity score matching.
• Figure 4: Selection of diagnostic density plots on key covariates between treated and control students.
• Figure 5: Density plots for selected baseline covariates (row 1).