Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)

TL;DR

This study benchmarks four state‑of‑the‑art LLMs (GPT‑4o, Claude 3.5, Llama 3.1, Gemini 1.5 Pro) on 28 items of the Master Interview Rating Scale (MIRS) across 10 OSCE cases to assess automated scoring of medical interview performance. Using zero‑shot, chain‑of‑thought, few‑shot, and multi‑step prompting, the authors report low exact accuracy ($\approx 0.27$–$0.52$) but moderate to high off‑by‑one ($\approx 0.67$–$0.91$) and thresholded ($\approx 0.75$–$0.91$) accuracy, with zero‑shot often performing best and item‑specific prompts sometimes improving results. A reliability check for GPT‑4o yielded $\alpha = 0.98$, indicating strong intra‑rater consistency, while multimodal nonverbal assessment with Gemini showed poor agreement ($\alpha = -0.47$), highlighting current limitations in applying AV‑capable models to nuanced clinical cues. The work provides a baseline benchmark for automated OSCE assessment, demonstrates feasibility for AI‑assisted scoring, and identifies directions for refining prompts, cross‑institution validation, and multimodal integration to support scalable feedback in medical education.

Abstract

Objective Structured Clinical Examinations (OSCEs) are widely used to assess medical students' communication skills, but scoring interview-based assessments is time-consuming and potentially subject to human bias. This study explored the potential of large language models (LLMs) to automate OSCE evaluations using the Master Interview Rating Scale (MIRS). We compared the performance of four state-of-the-art LLMs (GPT-4o, Claude 3.5, Llama 3.1, and Gemini 1.5 Pro) in evaluating OSCE transcripts across all 28 items of the MIRS under the conditions of zero-shot, chain-of-thought (CoT), few-shot, and multi-step prompting. The models were benchmarked against a dataset of 10 OSCE cases with 174 expert consensus scores available. Model performance was measured using three accuracy metrics (exact, off-by-one, thresholded). Averaging across all MIRS items and OSCE cases, LLMs performed with low exact accuracy (0.27 to 0.44), and moderate to high off-by-one accuracy (0.67 to 0.87) and thresholded accuracy (0.75 to 0.88). A zero temperature parameter ensured high intra-rater reliability (α = 0.98 for GPT-4o). CoT, few-shot, and multi-step techniques proved valuable when tailored to specific assessment items. The performance was consistent across MIRS items, independent of encounter phases and communication domains. We demonstrated the feasibility of AI-assisted OSCE evaluation and provided benchmarking of multiple LLMs across multiple prompt techniques. Our work provides a baseline performance assessment for LLMs that lays a foundation for future research into automated assessment of clinical communication skills.

Figures (4)

• Figure 1: The overall flow of the evaluation process (top), along with examples of an annotated transcript, MIRS item prompt, and score/justification pair (bottom). OSCE video transcripts are appended to MIRS item prompts, which are passed into LLMs for scoring and justification.
• Figure 2: Structure of the steps involved for each prompt, depending on the prompting technique, with one such prompt for each MIRS rubric item. In multi-step prompting, the “Provide Transcript” step uses the extracted excerpt.
• Figure 3: Average performance of each model using different prompting techniques and measured with different accuracy metrics. Error bars represent standard errors calculated across the 10 OSCE cases, where each case's accuracy is first computed as the mean of all applicable MIRS items for that case.
• Figure 4: Heatmap of the off-by-one accuracy for each model and MIRS item. While we analyzed a total of 26 text-based MIRS items, this figure displays results for the 21 items where data points were available across all cases. Opening the discussion and building the relationship correspond to the beginning of the visit. Information gathering and understanding of the patient's perspective occur primarily during the middle of the visit. Information sharing occurs throughout the visit. Reaching an agreement and providing closure correspond to the end of the visit.