Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Evaluation of Classroom Instructional Support with LLMs and BoWs: Connecting Global Predictions to Specific Feedback

Jacob Whitehill, Jennifer LoCasale-Crouch

TL;DR

The paper tackles the challenge of providing frequent, actionable feedback to teachers by automatically estimating CLASS Instructional Support from transcribed classroom dialogue. It compares zero-shot LLM-based utterance judgments (via Llama2) and a deterministic BoW baseline, aggregating utterance-level features with $L_1$-regularized regression to yield 15-minute global CLASS scores. Evaluations on two toddler/pre-K datasets show Pearson correlations up to $R=0.48$, approaching human inter-rater reliability, with LLMs usually outperforming BoW but gains when combining both. The approach yields interpretable, utterance-level explanations and visualizations that could support scalable, privacy-preserving feedback to teachers, while identifying challenges such as LLM hallucinations and the need for careful deployment in real-world settings.

Abstract

With the aim to provide teachers with more specific, frequent, and actionable feedback about their teaching, we explore how Large Language Models (LLMs) can be used to estimate ``Instructional Support'' domain scores of the CLassroom Assessment Scoring System (CLASS), a widely used observation protocol. We design a machine learning architecture that uses either zero-shot prompting of Meta's Llama2, and/or a classic Bag of Words (BoW) model, to classify individual utterances of teachers' speech (transcribed automatically using OpenAI's Whisper) for the presence of Instructional Support. Then, these utterance-level judgments are aggregated over a 15-min observation session to estimate a global CLASS score. Experiments on two CLASS-coded datasets of toddler and pre-kindergarten classrooms indicate that (1) automatic CLASS Instructional Support estimation accuracy using the proposed method (Pearson $R$ up to $0.48$) approaches human inter-rater reliability (up to $R=0.55$); (2) LLMs generally yield slightly greater accuracy than BoW for this task, though the best models often combined features extracted from both LLM and BoW; and (3) for classifying individual utterances, there is still room for improvement of automated methods compared to human-level judgments. Finally, (4) we illustrate how the model's outputs can be visualized at the utterance level to provide teachers with explainable feedback on which utterances were most positively or negatively correlated with specific CLASS dimensions.

Automated Evaluation of Classroom Instructional Support with LLMs and BoWs: Connecting Global Predictions to Specific Feedback

TL;DR

The paper tackles the challenge of providing frequent, actionable feedback to teachers by automatically estimating CLASS Instructional Support from transcribed classroom dialogue. It compares zero-shot LLM-based utterance judgments (via Llama2) and a deterministic BoW baseline, aggregating utterance-level features with -regularized regression to yield 15-minute global CLASS scores. Evaluations on two toddler/pre-K datasets show Pearson correlations up to , approaching human inter-rater reliability, with LLMs usually outperforming BoW but gains when combining both. The approach yields interpretable, utterance-level explanations and visualizations that could support scalable, privacy-preserving feedback to teachers, while identifying challenges such as LLM hallucinations and the need for careful deployment in real-world settings.

Abstract

With the aim to provide teachers with more specific, frequent, and actionable feedback about their teaching, we explore how Large Language Models (LLMs) can be used to estimate ``Instructional Support'' domain scores of the CLassroom Assessment Scoring System (CLASS), a widely used observation protocol. We design a machine learning architecture that uses either zero-shot prompting of Meta's Llama2, and/or a classic Bag of Words (BoW) model, to classify individual utterances of teachers' speech (transcribed automatically using OpenAI's Whisper) for the presence of Instructional Support. Then, these utterance-level judgments are aggregated over a 15-min observation session to estimate a global CLASS score. Experiments on two CLASS-coded datasets of toddler and pre-kindergarten classrooms indicate that (1) automatic CLASS Instructional Support estimation accuracy using the proposed method (Pearson up to ) approaches human inter-rater reliability (up to ); (2) LLMs generally yield slightly greater accuracy than BoW for this task, though the best models often combined features extracted from both LLM and BoW; and (3) for classifying individual utterances, there is still room for improvement of automated methods compared to human-level judgments. Finally, (4) we illustrate how the model's outputs can be visualized at the utterance level to provide teachers with explainable feedback on which utterances were most positively or negatively correlated with specific CLASS dimensions.
Paper Structure (25 sections, 1 equation, 2 figures, 3 tables)

This paper contains 25 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CLassroom Assessment Scoring System (CLASS)
  4. AI for Automated Classroom Analysis
  5. Architecture
  6. Datasets
  7. Preprocessing
  8. Inter-Rater Reliability
  9. Method I: Large Language Models (LLM)
  10. Method II: Bag of Words (BoW)
  11. Feature Aggregation and $L_1$ Regression to Estimate CLASS Scores
  12. Experiments
  13. Results
  14. Analysis of Llama2's Judgments of Individual Utterances
  15. Results
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Our high-level approach for CLASS score estimation: After transcribing each classroom audio into text, each utterance is analyzed using either Llama2 or a BoW model to produce a feature vector. The feature vectors are then aggregated (summed and z-scored) across utterances, and finally regressed into a CLASS score estimate.
  • Figure 2: Temporal heatmap of CLASS "Instructional Support" dimension. The top 4 most and least highly related utterances from the classroom audio are highlighted.