How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discourse

TL;DR

This work evaluates how well out-of-the-box foundation models interpret authentic instructional discourse by identifying Talk Moves in classroom transcripts. It benchmarks six leading systems under zero-, one-, and few-shot prompting against expert annotations, using Cohen's $\kappa$ to quantify agreement. Results show moderate reliability ($\kappa$ in $0.38$–$0.58$) with notable variability across moves and models, though extensive in-context prompting can substantially boost performance for some models. The findings underscore latent conversational reasoning capabilities in current AI while highlighting limits that motivate hybrid AI–human annotation workflows for discourse-rich educational tasks.

Abstract

Large language models (LLMs) are increasingly adopted in educational technologies for a variety of tasks, from generating instructional materials and assisting with assessment design to tutoring. While prior work has investigated how models can be adapted or optimized for specific tasks, far less is known about how well LLMs perform at interpreting authentic educational scenarios without significant customization. As LLM-based systems become widely adopted by learners and educators in everyday academic contexts, understanding their out-of-the-box capabilities is increasingly important for setting expectations and benchmarking. We compared six LLMs to estimate their baseline performance on a simple but important task: classifying instructional moves in authentic classroom transcripts. We evaluated typical prompting methods: zero-shot, one-shot, and few-shot prompting. We found that while zero-shot performance was moderate, providing comprehensive examples (few-shot prompting) significantly improved performance for state-of-the-art models, with the strongest configuration reaching Cohen's Kappa = 0.58 against expert-coded annotations. At the same time, improvements were neither uniform nor complete: performance varied considerably by instructional move, and higher recall frequently came at the cost of increased false positives. Overall, these findings indicate that foundation models demonstrate meaningful yet limited capacity to interpret instructional discourse, with prompt design helping to surface capability but not eliminating fundamental reliability constraints.

Figures (2)

• Figure 1: Per-Code Cohen's $\kappa$ by Prompt Strategy and Model. Bars represent mean $\kappa$; points show individual model performance with 95% confidence intervals.
• Figure 2: Kappa scores for each model across zero-shot, one-shot, and few-shot prompting conditions. Error bars represent 95% bootstrapped confidence intervals. Brackets indicate statistically significant pairwise differences determined by paired bootstrap test ($R=1,000$). Note: Overlapping 95% CIs do not preclude significant differences when using paired testing.