Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Not All Students Engage Alike: Multi-Institution Patterns in GenAI Tutor Use

Youjie Chen, Xixi Shi, Xinyu Liu, Shuaiguo Wang, Tracy Xiao Liu, Dragan Gašević

TL;DR

This study addresses how students engage with GenAI Tutors in authentic postsecondary settings by analyzing de-identified LMS and tutor interaction logs across 10 institutions and 200 classes. It introduces a two-stage analytic pipeline that first segments conversation sessions and then applies process mining to uncover evolving student engagement patterns, identifying four session-level types and four student-level cycles. Engagement varies systematically by institutional selectivity and course discipline, with deep and routine-learning use more common in highly selective institutions and STEM courses showing more shallow engagement. The findings offer a framework for analyzing human-AI interactions at scale and highlight discipline-sensitive design and equity considerations for responsible deployment of GenAI Tutors in higher education.

Abstract

The emergence of generative artificial intelligence (GenAI) has created unprecedented opportunities to provide individualized learning support in classrooms as automated tutoring systems at scale. However, concerns have been raised that students may engage with these tools in ways that do not support learning. Moreover, student engagement with GenAI Tutors may vary across instructional contexts, potentially leading to unequal learning experiences. In this study, we utilize de-identified student interaction logs from an existing GenAI Tutor and the learning management system in which it is embedded. We systematically examined student engagement (N = 11,406) with the tool across 200 classes in ten post-secondary institutions through a two-stage pipeline: First, we identified four distinct engagement types at the conversation session level. In particular, 10.4% of them were "shallow engagement" where copy-pasting behavior was prevalent. Then, at the student level, we show that students transitioned across engagement types over time. However, students who exhibited shallow engagement with the tool were more likely to remain in this mode, whereas those who engaged deeply with the tool transitioned more flexibly across engagement types. Finally, at both the session and student levels, we show substantial heterogeneity in student engagement across institution selectivity and course disciplines. In particular, students from highly selective institutions were more likely to exhibit deep engagement. Together, our study advances the understanding of how GenAI Tutors are used in authentic educational settings and provides a framework for analyzing student engagement with GenAI Tutors, with implications for responsible implementation at scale.

Not All Students Engage Alike: Multi-Institution Patterns in GenAI Tutor Use

TL;DR

This study addresses how students engage with GenAI Tutors in authentic postsecondary settings by analyzing de-identified LMS and tutor interaction logs across 10 institutions and 200 classes. It introduces a two-stage analytic pipeline that first segments conversation sessions and then applies process mining to uncover evolving student engagement patterns, identifying four session-level types and four student-level cycles. Engagement varies systematically by institutional selectivity and course discipline, with deep and routine-learning use more common in highly selective institutions and STEM courses showing more shallow engagement. The findings offer a framework for analyzing human-AI interactions at scale and highlight discipline-sensitive design and equity considerations for responsible deployment of GenAI Tutors in higher education.

Abstract

The emergence of generative artificial intelligence (GenAI) has created unprecedented opportunities to provide individualized learning support in classrooms as automated tutoring systems at scale. However, concerns have been raised that students may engage with these tools in ways that do not support learning. Moreover, student engagement with GenAI Tutors may vary across instructional contexts, potentially leading to unequal learning experiences. In this study, we utilize de-identified student interaction logs from an existing GenAI Tutor and the learning management system in which it is embedded. We systematically examined student engagement (N = 11,406) with the tool across 200 classes in ten post-secondary institutions through a two-stage pipeline: First, we identified four distinct engagement types at the conversation session level. In particular, 10.4% of them were "shallow engagement" where copy-pasting behavior was prevalent. Then, at the student level, we show that students transitioned across engagement types over time. However, students who exhibited shallow engagement with the tool were more likely to remain in this mode, whereas those who engaged deeply with the tool transitioned more flexibly across engagement types. Finally, at both the session and student levels, we show substantial heterogeneity in student engagement across institution selectivity and course disciplines. In particular, students from highly selective institutions were more likely to exhibit deep engagement. Together, our study advances the understanding of how GenAI Tutors are used in authentic educational settings and provides a framework for analyzing student engagement with GenAI Tutors, with implications for responsible implementation at scale.
Paper Structure (20 sections, 7 figures, 1 table)

This paper contains 20 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Evolution of Automated Tutoring Systems
  4. Student Engagement with GenAI Tutors
  5. Learning Analytics to Reveal GenAI Interaction Patterns
  6. Methods
  7. Study Context
  8. Data
  9. Analytical Approach
  10. Identifying and Segmenting Conversation Sessions
  11. Feature Engineering
  12. Session-Level Clustering Analysis
  13. Student-Level Process Mining
  14. Results
  15. Descriptive Results of Student Engagement
  16. ...and 5 more sections

Figures (7)

  • Figure 1: Monthly Trends of Student Engagement with the GenAI Tutor from Mid-February (Semester Starts) to Mid-July (Semester Ends). Panel (a) shows the number of students who used the GenAI Tutor, and Panel (b) shows the average number of conversation sessions among students who used the GenAI Tutor.
  • Figure 2: Heatmap of Cluster Centroids across Features. Each cell shows the standardized score (z-score) indicating how much each cluster's mean deviates from the overall mean.
  • Figure 3: Distribution of Conversation Sessions across Academic Weeks by Engagement Type
  • Figure 4: Distribution of Session-level Engagement Types by Institution Selectivity (a) and Course Discipline (b). Error bars show 95% CIs with cluster-robust SEs.
  • Figure 5: Transition Probability Matrix of Student Engagement Types. Each cell reports the conditional probability of transitioning from state $i$ (row) to state $j$ (column), with the absolute number of sequences shown in parentheses.
  • ...and 2 more figures