AI Annotation Orchestration: Evaluating LLM verifiers to Improve the Quality of LLM Annotations in Learning Analytics

TL;DR

The paper tackles the reliability problems of LLM-based tutoring annotations by introducing verification-oriented orchestration, including self-verification and cross-verification, and evaluating it across three frontier LLMs on authentic 1:1 math tutoring transcripts. Using a ground-truth adjudication process, the study demonstrates that self-verification substantially improves Cohen's $\kappa$—especially for intent-sensitive moves—while cross-verification offers additional, but conditional, gains depending on model pairing and construct. The authors provide a flexible orchestration framework, empirical cross-LLM comparisons, and a standardized verifier(annotator) notation to enable replication and transparent reporting. Collectively, the work argues that verification-driven pipelines yield more reliable, scalable, and auditable LLM-assisted tutoring analytics, with clear guidance on when to employ self-versus cross-verification and when to involve human adjudication. These findings have practical implications for scaling qualitative coding in learning analytics and related education research domains.

Abstract

Large Language Models (LLMs) are increasingly used to annotate learning interactions, yet concerns about reliability limit their utility. We test whether verification-oriented orchestration-prompting models to check their own labels (self-verification) or audit one another (cross-verification)-improves qualitative coding of tutoring discourse. Using transcripts from 30 one-to-one math sessions, we compare three production LLMs (GPT, Claude, Gemini) under three conditions: unverified annotation, self-verification, and cross-verification across all orchestration configurations. Outputs are benchmarked against a blinded, disagreement-focused human adjudication using Cohen's kappa. Overall, orchestration yields a 58 percent improvement in kappa. Self-verification nearly doubles agreement relative to unverified baselines, with the largest gains for challenging tutor moves. Cross-verification achieves a 37 percent improvement on average, with pair- and construct-dependent effects: some verifier-annotator pairs exceed self-verification, while others reduce alignment, reflecting differences in verifier strictness. We contribute: (1) a flexible orchestration framework instantiating control, self-, and cross-verification; (2) an empirical comparison across frontier LLMs on authentic tutoring data with blinded human "gold" labels; and (3) a concise notation, verifier(annotator) (e.g., Gemini(GPT) or Claude(Claude)), to standardize reporting and make directional effects explicit for replication. Results position verification as a principled design lever for reliable, scalable LLM-assisted annotation in Learning Analytics.

Figures (3)

• Figure 1: Reliability of Unverified LLM Annotations by Tutor–Move Category. Bars show Cohen’s $\kappa$ with ground truth for Claude, Gemini, and GPT.
• Figure 2: Improvement in $\kappa$ from the unverified baseline LLM annotation of the corresponding annotator model (e.g., the $\kappa$ improvement for GPT(Gemini) here is relative to Gemini). Each point represents a verifier(annotator) combination; markers above zero indicate improved agreement, while negative values reflect declines.
• Figure 3: Per–model reliability under different verification strategies. Each facet fixes the base annotator; points show Cohen’s $\kappa$ across categories for baseline, self–verification, and cross–verification directions. Yellow lines denote means.