Leveraging Large Language Models to Identify Conversation Threads in Collaborative Learning

TL;DR

This paper develops and validates a systematic threading framework for synchronous, multi-party conversation in collaborative learning and investigates whether explicit thread linkages improve large language model (LLM) performance on downstream discourse coding. It introduces a human-grounded guidebook for threading, benchmarks multiple LLM prompting strategies for identifying threads, and empirically tests how threading affects downstream coding within the ABCDE framework. Key findings show that providing clear conversational thread information improves LLM coding accuracy, with sliding-window prompts and LLM-generated threading offering favorable trade-offs between performance, time, and cost. The work highlights the importance of structured dialogue representations for scalable, real-time analysis of complex group interactions and discusses practical paths toward hybrid human-AI approaches to maximize value.

Abstract

Understanding how ideas develop and flow in small-group conversations is critical for analyzing collaborative learning. A key structural feature of these interactions is threading, the way discourse talk naturally organizes into interwoven topical strands that evolve over time. While threading has been widely studied in asynchronous text settings, detecting threads in synchronous spoken dialogue remains challenging due to overlapping turns and implicit cues. At the same time, large language models (LLMs) show promise for automating discourse analysis but often struggle with long-context tasks that depend on tracing these conversational links. In this paper, we investigate whether explicit thread linkages can improve LLM-based coding of relational moves in group talk. We contribute a systematic guidebook for identifying threads in synchronous multi-party transcripts and benchmark different LLM prompting strategies for automated threading. We then test how threading influences performance on downstream coding of conversational analysis frameworks, that capture core collaborative actions such as agreeing, building, and eliciting. Our results show that providing clear conversational thread information improves LLM coding performance and underscores the heavy reliance of downstream analysis on well-structured dialogue. We also discuss practical trade-offs in time and cost, emphasizing where human-AI hybrid approaches can yield the best value. Together, this work advances methods for combining LLMs and robust conversational thread structures to make sense of complex, real-time group interactions.

Figures (4)

• Figure 1: Anatomy of a Thread
• Figure 2: Activities performed by students in the data science workshop spanning our dataset
• Figure 3: Illustrative examples of conversation threads showing overlaps, multiple contributions within a single turn, consensus, backchannels, and transitions.
• Figure 4: Time/Cost vs. Performance Tradeoff for average of categories with best interrator reliability E: Eliciting responses or actions. Performance: Left: Time vs. Cohen's Kappa for Right: Monetary Cost vs. Cohen's Kappa