Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Meta-Analysis of LLM Effects on Students across Qualification, Socialisation, and Subjectification

Jiayu Huang, Ruoxin Ritter Wang, Jen-Hao Liu, Boming Xia, Yue Huang, Ruoxi Sun, Jason Minhui Xue, Jinan Zou

TL;DR

This study reframes LLMs in education through Biesta's three purposes—qualification, socialisation, and subjectification—and conducts a large-scale meta-analysis of 133 experimental and quasi-experimental studies (k = 188) to test how intervention design shapes outcomes. Across all domains, LLMs show positive effects, with the strongest and most reliable gains in qualification when LLMs serve as sustained tutors. Socialisation and subjectification show positive but more context-dependent gains, hinging on long-term, structured pedagogy and small-group, autonomy-focused designs. The findings stress that design decisions, not just access, determine educational impact, with important implications for theory, evaluation methods, classroom practice, and policy toward fostering agency, participation, and durable learning in AI-enhanced education.

Abstract

Large language models (LLMs) are increasingly positioned as solutions for education, yet evaluations often reduce their impact to narrow performance metrics. This paper reframes the question by asking "what kind of impact should LLMs have in education?" Drawing on Biesta's tripartite account of good education: qualification, socialisation, and subjectification, we present a meta-analysis of 133 experimental and quasi-experimental studies (k = 188). Overall, the impact of LLMs on student learning is positive but uneven. Strong effects emerge in qualification, particularly when LLMs function as tutors in sustained interventions. Socialisation outcomes appear more variable, concentrated in sustained, reflective interventions. Subjectification, linked to autonomy and learner development, remains fragile, with improvements confined to small-scale, long-term studies. This purpose-level view highlights design as the decisive factor: without scaffolds for participation and agency, LLMs privilege what is easiest to measure while neglecting broader aims of education. For HCI and education, the issue is not just whether LLMs work, but what futures they enable or foreclose.

A Meta-Analysis of LLM Effects on Students across Qualification, Socialisation, and Subjectification

TL;DR

This study reframes LLMs in education through Biesta's three purposes—qualification, socialisation, and subjectification—and conducts a large-scale meta-analysis of 133 experimental and quasi-experimental studies (k = 188) to test how intervention design shapes outcomes. Across all domains, LLMs show positive effects, with the strongest and most reliable gains in qualification when LLMs serve as sustained tutors. Socialisation and subjectification show positive but more context-dependent gains, hinging on long-term, structured pedagogy and small-group, autonomy-focused designs. The findings stress that design decisions, not just access, determine educational impact, with important implications for theory, evaluation methods, classroom practice, and policy toward fostering agency, participation, and durable learning in AI-enhanced education.

Abstract

Large language models (LLMs) are increasingly positioned as solutions for education, yet evaluations often reduce their impact to narrow performance metrics. This paper reframes the question by asking "what kind of impact should LLMs have in education?" Drawing on Biesta's tripartite account of good education: qualification, socialisation, and subjectification, we present a meta-analysis of 133 experimental and quasi-experimental studies (k = 188). Overall, the impact of LLMs on student learning is positive but uneven. Strong effects emerge in qualification, particularly when LLMs function as tutors in sustained interventions. Socialisation outcomes appear more variable, concentrated in sustained, reflective interventions. Subjectification, linked to autonomy and learner development, remains fragile, with improvements confined to small-scale, long-term studies. This purpose-level view highlights design as the decisive factor: without scaffolds for participation and agency, LLMs privilege what is easiest to measure while neglecting broader aims of education. For HCI and education, the issue is not just whether LLMs work, but what futures they enable or foreclose.

Paper Structure

This paper contains 31 sections, 1 equation, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Student-Facing LLMs: Empirical Baseline
  4. Prior Syntheses: Coverage and Methods
  5. Design Moderating Factors in Prior Work
  6. Theoretical Framework
  7. Methodology
  8. Search Strategy
  9. Inclusion and Exclusion Criteria
  10. Data Extraction and Coding
  11. Statistical Analysis
  12. Results
  13. Descriptive Statistics: A Rapidly Growing but Skewed Research Landscape
  14. Overall Effects: LLMs Enhance Learning, but Unevenly Across Domains
  15. Moderator Analyses: When and How LLMs Work
  16. ...and 16 more sections

Figures (4)

  • Figure 1: An overview of our meta-analysis framework.
  • Figure 2: Funnel plot of studies on Qualification (95% CI).
  • Figure 3: Funnel plot of studies on Socialisation (95% CI).
  • Figure 4: Funnel plot of studies on Subjectification (95% CI).