Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis

TL;DR

The paper addresses how to characterize synergistic learning in students' collaborative discourse by leveraging human-in-the-loop prompting with GPT-4-Turbo. It extends Chain-of-Thought Prompting + Active Learning to produce context-rich summaries and categorize discourse into physics-focused, computing-focused, synergistic, or separate interactions within the C2STEM kinematics task. In a small, exploratory study, GPT-4-Turbo approaches human performance in generating useful, detailed analyses while GPT-4 occasionally hallucinates, underscoring both promise and limitations of AI-assisted classroom feedback. The work demonstrates potential for scalable, actionable insights to support teachers in guiding cross-domain learning, while outlining clear directions for larger-scale validation across tasks.

Abstract

LLMs have demonstrated proficiency in contextualizing their outputs using human input, often matching or beating human-level performance on a variety of tasks. However, LLMs have not yet been used to characterize synergistic learning in students' collaborative discourse. In this exploratory work, we take a first step towards adopting a human-in-the-loop prompt engineering approach with GPT-4-Turbo to summarize and categorize students' synergistic learning during collaborative discourse. Our preliminary findings suggest GPT-4-Turbo may be able to characterize students' synergistic learning in a manner comparable to humans and that our approach warrants further investigation.

Figures (1)

• Figure 1: Chain-of-Thought Prompting + Active Learning, identified by the green box, where each blue diamond is a step in the Method. Yellow boxes represent the process's application to the classroom detailed in prior work cohn2024chain.