CaseMaster: Designing and Evaluating a Probe for Oral Case Presentation Training with LLM Assistance

TL;DR

This work tackles the challenge of training medical students in oral case presentations (OCP) by integrating large language models (LLMs) into a structured, two-stage training probe called CaseMaster. Through a formative study with six educators, the authors derive design concepts and goals, then implement CaseMaster as a web-based tool that guides Preparation (case exploration and SOAP drafting) and Reflection (comparison to references and LLMed scoring). In a controlled study with 12 students, CaseMaster improved differential-diagnosis clarity and showed favorable usability trends versus a baseline, while expert evaluation confirmed perceived benefits and highlighted areas for transparency and customization. The paper concludes with design guidelines for reliable, pedagogically grounded, and adaptable LLM-supported training in medical education, emphasizing balanced integration with traditional instruction and attention to ethics and privacy. The findings suggest CaseMaster can streamline OCP training, mitigate workload, and provide a blueprint for scalable LLM-enabled medical education tools.

Abstract

Preparing an oral case presentation (OCP) is a crucial skill for medical students, requiring clear communication of patient information, clinical findings, and treatment plans. However, inconsistent student participation and limited guidance can make this task challenging. While Large Language Models (LLMs) can provide structured content to streamline the process, their role in facilitating skill development and supporting medical education integration remains underexplored. To address this, we conducted a formative study with six medical educators and developed CaseMaster, an interactive probe that leverages LLM-generated content tailored to medical education to help users enhance their OCP skills. The controlled study suggests CaseMaster has the potential to both improve presentation quality and reduce workload compared to traditional methods, an implication reinforced by expert feedback. We propose guidelines for educators to develop adaptive, user-centered training methods using LLMs, while considering the implications of integrating advanced technologies into medical education.

Figures (10)

• Figure 1: Example of an oral case presentation: On the left, a group of healthcare professionals, including doctors and medical students, are gathering in a clinical hallway for a case presentation, with the presenter leading the discussion while the others listen attentively. On the right, the case presentation is structured into four key sections: Orange (Subjective) for history and symptoms, Blue (Objective) for findings, Purple (Assessment) for diagnosis, and Pink (Plan) for treatment.
• Figure 2: Co-design session focusing on design concepts: participants discussed OCP practices, manipulated sketches to explore ideas, and reflected to refine designs in real time. The design concepts supported both the Preparation and Reflection stages, allowing flexible, low-fidelity exploration and iterative design refinement.
• Figure 3: During the Preparation Stage, users can choose a patient from the (A) Patient List, review detailed information in the (B) Patient Record, and draft and refine their presentation within the (C) Preparation Panel. The Preparation Panel includes LLM-powered assistance for content evaluation, suggestions, and generating specific examples to support users in crafting their presentation.
• Figure 4: During the Reflection Stage, users can compare their solutions in the (D) Validation panel, where highlighted discrepancies emphasize the differences between their solutions and the reference. The (E) Score Sheet displays LLM-generated scores across various evaluation criteria, with clickable red checkmarks offering detailed feedback.
• Figure 5: The procedure for the user study comparing CaseMaster with a baseline system was conducted remotely via Zoom. Participants completed two OCP tasks using pre-study materials, followed by independent preparation, completion of feedback questionnaires, and a final interview discussion.