Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ensembling Large Language Models to Characterize Affective Dynamics in Student-AI Tutor Dialogues

Chenyu Zhang, Sharifa Alghowinem, Cynthia Breazeal

TL;DR

This paper addresses the gap in understanding how learners’ affective states evolve during LLM‑mediated tutoring. It introduces an ensemble affect-sensing framework that fuses zero-shot predictions from Gemini, GPT‑4o, and Claude to produce turn-level VAL scores and consensus emotion labels, using rank-weighted pooling and plurality consensus. Temporal analysis via a first‑order Markov model reveals positive affect as relatively persistent, with frequent but short-lived shifts to neutral or negative states and a neutral gateway that can steer toward positive outcomes. The findings highlight the nuanced affective landscape in AI tutoring and argue for affect-aware interventions to enhance learning, while outlining limitations and directions for future validation and methodological refinement.

Abstract

While recent studies have examined the leaning impact of large language model (LLM) in educational contexts, the affective dynamics of LLM-mediated tutoring remain insufficiently understood. This work introduces the first ensemble-LLM framework for large-scale affect sensing in tutoring dialogues, advancing the conversation on responsible pathways for integrating generative AI into education by attending to learners' evolving affective states. To achieve this, we analyzed two semesters' worth of 16,986 conversational turns exchanged between PyTutor, an LLM-powered AI tutor, and 261 undergraduate learners across three U.S. institutions. To investigate learners' emotional experiences, we generate zero-shot affect annotations from three frontier LLMs (Gemini, GPT-4o, Claude), including scalar ratings of valence, arousal, and learning-helpfulness, along with free-text emotion labels. These estimates are fused through rank-weighted intra-model pooling and plurality consensus across models to produce robust emotion profiles. Our analysis shows that during interaction with the AI tutor, students typically report mildly positive affect and moderate arousal. Yet learning is not uniformly smooth: confusion and curiosity are frequent companions to problem solving, and frustration, while less common, still surfaces in ways that can derail progress. Emotional states are short-lived--positive moments last slightly longer than neutral or negative ones, but they are fragile and easily disrupted. Encouragingly, negative emotions often resolve quickly, sometimes rebounding directly into positive states. Neutral moments frequently act as turning points, more often steering students upward than downward, suggesting opportunities for tutors to intervene at precisely these junctures.

Ensembling Large Language Models to Characterize Affective Dynamics in Student-AI Tutor Dialogues

TL;DR

This paper addresses the gap in understanding how learners’ affective states evolve during LLM‑mediated tutoring. It introduces an ensemble affect-sensing framework that fuses zero-shot predictions from Gemini, GPT‑4o, and Claude to produce turn-level VAL scores and consensus emotion labels, using rank-weighted pooling and plurality consensus. Temporal analysis via a first‑order Markov model reveals positive affect as relatively persistent, with frequent but short-lived shifts to neutral or negative states and a neutral gateway that can steer toward positive outcomes. The findings highlight the nuanced affective landscape in AI tutoring and argue for affect-aware interventions to enhance learning, while outlining limitations and directions for future validation and methodological refinement.

Abstract

While recent studies have examined the leaning impact of large language model (LLM) in educational contexts, the affective dynamics of LLM-mediated tutoring remain insufficiently understood. This work introduces the first ensemble-LLM framework for large-scale affect sensing in tutoring dialogues, advancing the conversation on responsible pathways for integrating generative AI into education by attending to learners' evolving affective states. To achieve this, we analyzed two semesters' worth of 16,986 conversational turns exchanged between PyTutor, an LLM-powered AI tutor, and 261 undergraduate learners across three U.S. institutions. To investigate learners' emotional experiences, we generate zero-shot affect annotations from three frontier LLMs (Gemini, GPT-4o, Claude), including scalar ratings of valence, arousal, and learning-helpfulness, along with free-text emotion labels. These estimates are fused through rank-weighted intra-model pooling and plurality consensus across models to produce robust emotion profiles. Our analysis shows that during interaction with the AI tutor, students typically report mildly positive affect and moderate arousal. Yet learning is not uniformly smooth: confusion and curiosity are frequent companions to problem solving, and frustration, while less common, still surfaces in ways that can derail progress. Emotional states are short-lived--positive moments last slightly longer than neutral or negative ones, but they are fragile and easily disrupted. Encouragingly, negative emotions often resolve quickly, sometimes rebounding directly into positive states. Neutral moments frequently act as turning points, more often steering students upward than downward, suggesting opportunities for tutors to intervene at precisely these junctures.
Paper Structure (10 sections, 2 figures, 1 table)

This paper contains 10 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Dataset
  4. Automatic Emotion Annotation & Ensemble Fusion
  5. Temporal Analyses
  6. Results
  7. Descriptive Affective Landscape
  8. Temporal & Transition Dynamics
  9. Discussion
  10. Conclusion

Figures (2)

  • Figure 1: Affective profile of PyTutor student turns.
  • Figure 2: Markov transition matrix ($3\times3$) for valence tertiles (student turns only; global tertiles; $\beta\!=\!1$ smoothing). Row $=$ current state, column $=$ next state.