Measuring Teaching with LLMs

TL;DR

This work tackles the persistent challenge of measuring teaching quality by introducing custom LLMs built on sentence-level embeddings to score $25$ instructional dimensions in long classroom transcripts. It deploys a data-efficient, multitask encoder framework, evaluating multiple sentence-embedding models and demonstrating strong alignment with expert ratings—often reaching human-level performance and surpassing average human-human correlations. The study also reveals that mature models shift variance toward lesson-level features, challenging single-turn evaluation paradigms, and validates external validity by linking aggregate ratings to teacher value-added measures, though item-level generalization remains limited. Despite high performance, the authors urge caution for deployment, highlighting data and domain limitations and advocating a human-in-the-loop approach for scalable, reliable feedback in educator development.

Abstract

Objective and scalable measurement of teaching quality is a persistent challenge in education. While Large Language Models (LLMs) offer potential, general-purpose models have struggled to reliably apply complex, authentic classroom observation instruments. This paper uses custom LLMs built on sentence-level embeddings, an architecture better suited for the long-form, interpretive nature of classroom transcripts than conventional subword tokenization. We systematically evaluate five different sentence embeddings under a data-efficient training regime designed to prevent overfitting. Our results demonstrate that these specialized models can achieve human-level and even super-human performance with expert human ratings above 0.65 and surpassing the average human-human rater correlation. Further, through analysis of annotation context windows, we find that more advanced models-those better aligned with human judgments-attribute a larger share of score variation to lesson-level features rather than isolated utterances, challenging the sufficiency of single-turn annotation paradigms. Finally, to assess external validity, we find that aggregate model scores align with teacher value-added measures, indicating they are capturing features relevant to student learning. However, this trend does not hold at the individual item level, suggesting that while the models learn useful signals, they have not yet achieved full generalization. This work establishes a viable and powerful new methodology for AI-driven instructional measurement, offering a path toward providing scalable, reliable, and valid feedback for educator development.

Figures (12)

• Figure 1: Correlation with human experts across training epochs. The MQI instrument had at least two human raters per lesson, and the mean and interquartile range of all 63 human MQI raters correlated across the other raters are represented by the gray line and shaded region in the figure.
• Figure 2: Proportion of variation explained as related to a model's alignment to human expert ratings.
• Figure 3: $\tau$-canonical correlation between classroom observation ratings and value-added measures as a function of model alignment to human expert ratings.
• Figure 4: Human Expert Score Distributions. These are the score distributions from human experts. The distinct rating patterns highlight the underlying qualitative differences in the constructs being rated. Previous studies have focused on a limited range of items (bolded, wang_is_2023)
• Figure 5: Held-out Test Set Distributions. These are comparative score distributions from human experts for the items in the held-out test set and the remaining sample. No differences were statistically significant.