LLM-Assisted Automated Deductive Coding of Dialogue Data: Leveraging Dialogue-Specific Characteristics to Enhance Contextual Understanding
Ying Na, Shihui Feng
TL;DR
The paper tackles the contextual understanding challenges of large language models in dialogue data by introducing a contextually consistent LLM-assisted dialogue coding framework that separately predicts Communicative Events and Communicative Acts. It leverages role prompts, chain-of-thought reasoning, and a three-model ensemble to generate predictions, followed by a consistency checking stage that uses interdependencies between acts and events to improve accuracy. The approach demonstrates that separating event and act predictions, using multi-LLM collaboration, and applying iterative consistency checks yields higher agreement and reliability, with act predictions consistently outperforming event predictions and strong performance on a test set. The framework offers a scalable, domain-specific method for automated dialogue coding in education and points to multimodal data integration as a valuable future direction.
Abstract
Dialogue data has been a key source for understanding learning processes, offering critical insights into how students engage in collaborative discussions and how these interactions shape their knowledge construction. The advent of Large Language Models (LLMs) has introduced promising opportunities for advancing qualitative research, particularly in the automated coding of dialogue data. However, the inherent contextual complexity of dialogue presents unique challenges for these models, especially in understanding and interpreting complex contextual information. This study addresses these challenges by developing a novel LLM-assisted automated coding approach for dialogue data. The novelty of our proposed framework is threefold: 1) We predict the code for an utterance based on dialogue-specific characteristics -- communicative acts and communicative events -- using separate prompts following the role prompts and chain-of-thoughts methods; 2) We engaged multiple LLMs including GPT-4-turbo, GPT-4o, DeepSeek in collaborative code prediction; 3) We leveraged the interrelation between events and acts to implement consistency checking using GPT-4o. In particular, our contextual consistency checking provided a substantial accuracy improvement. We also found the accuracy of act predictions was consistently higher than that of event predictions. This study contributes a new methodological framework for enhancing the precision of automated coding of dialogue data as well as offers a scalable solution for addressing the contextual challenges inherent in dialogue analysis.