ChatGPT for automated grading of short answer questions in mechanical ventilation
Tejas Jade, Alex Yartsev
TL;DR
The study investigates ChatGPT 4o as an automated grader for short-answer questions in a postgraduate medical course on mechanical ventilation. It uses a retrospective audit of 557 responses from 215 students with a standardized rubric, comparing AI grades to human scoring across multiple AI prompts and sessions analyzed via G-theory, ICCs, and Bland-Altman methods. Results reveal systematic AI undergrading and poor individual-level agreement, with substantial discrepancies for certain rubric types, indicating current consumer LLMs are not ready for high-stakes postgraduate assessment without further validation. The findings highlight the potential for improvement through prompt engineering and rubric design, while cautioning against deployment in high-stakes settings without further validation.
Abstract
Standardised tests using short answer questions (SAQs) are common in postgraduate education. Large language models (LLMs) simulate conversational language and interpret unstructured free-text responses in ways aligning with applying SAQ grading rubrics, making them attractive for automated grading. We evaluated ChatGPT 4o to grade SAQs in a postgraduate medical setting using data from 215 students (557 short-answer responses) enrolled in an online course on mechanical ventilation (2020--2024). Deidentified responses to three case-based scenarios were presented to ChatGPT with a standardised grading prompt and rubric. Outputs were analysed using mixed-effects modelling, variance component analysis, intraclass correlation coefficients (ICCs), Cohen's kappa, Kendall's W, and Bland--Altman statistics. ChatGPT awarded systematically lower marks than human graders with a mean difference (bias) of -1.34 on a 10-point scale. ICC values indicated poor individual-level agreement (ICC1 = 0.086), and Cohen's kappa (-0.0786) suggested no meaningful agreement. Variance component analysis showed minimal variability among the five ChatGPT sessions (G-value = 0.87), indicating internal consistency but divergence from the human grader. The poorest agreement was observed for evaluative and analytic items, whereas checklist and prescriptive rubric items had less disagreement. We caution against the use of LLMs in grading postgraduate coursework. Over 60% of ChatGPT-assigned grades differed from human grades by more than acceptable boundaries for high-stakes assessments.