Disagreement as Data: Reasoning Trace Analytics in Multi-Agent Systems

TL;DR

This paper investigates whether reasoning traces produced by LLM agents in a multi-agent coding framework can serve as a new form of process data for learning analytics. It proposes using cosine similarity on reasoning traces to detect and interpret disagreements among agents and interpret them as informative analytic signals. An analysis of $9,746$ agent pairs on tutoring dialogues shows that reasoning similarity tracks code agreement and relates to human coding reliability, with robust patterns across conditions. The work demonstrates how combining quantitative reasoning-trace metrics with qualitative review can refine codebooks, surface ambiguity, and support more reliable, scalable human–AI qualitative coding in education.

Abstract

Learning analytics researchers often analyze qualitative student data such as coded annotations or interview transcripts to understand learning processes. With the rise of generative AI, fully automated and human-AI workflows have emerged as promising methods for analysis. However, methodological standards to guide such workflows remain limited. In this study, we propose that reasoning traces generated by large language model (LLM) agents, especially within multi-agent systems, constitute a novel and rich form of process data to enhance interpretive practices in qualitative coding. We apply cosine similarity to LLM reasoning traces to systematically detect, quantify, and interpret disagreements among agents, reframing disagreement as a meaningful analytic signal. Analyzing nearly 10,000 instances of agent pairs coding human tutoring dialog segments, we show that LLM agents' semantic reasoning similarity robustly differentiates consensus from disagreement and correlates with human coding reliability. Qualitative analysis guided by this metric reveals nuanced instructional sub-functions within codes and opportunities for conceptual codebook refinement. By integrating quantitative similarity metrics with qualitative review, our method has the potential to improve and accelerate establishing inter-rater reliability during coding by surfacing interpretive ambiguity, especially when LLMs collaborate with humans. We discuss how reasoning-trace disagreements represent a valuable new class of analytic signals advancing methodological rigor and interpretive depth in educational research.

Figures (1)

• Figure 1: Validation of cosine similarity as a metric for reasoning alignment. (a) The metric is robust, as the distinction between agreement and disagreement pairs remains stable across temperature settings. (b) The metric is sensitive, with distributions revealing interpretive ambiguity across different code categories.