Intelligent Documentation in Medical Education: Can AI Replace Manual Case Logging?

TL;DR

This study evaluates the feasibility of using large language models to automate procedural case log documentation in radiology education by converting free-text radiology reports into structured entries. Across six instruction-tuned models evaluated in a HIPAA-compliant, zero-shot setup, prompting strategies substantially outperform a metadata-only baseline, with best F1-scores near 0.87. The results reveal trade-offs between accuracy, latency, and cost for local versus commercial deployments, and demonstrate notable gains in sensitivity when chain-of-thought prompting is employed. The findings support AI-assisted documentation as a promising approach to reduce clerical burden and improve consistency, while underscoring the need for multi-institutional validation and careful human-in-the-loop integration before clinical deployment.

Abstract

Procedural case logs are a core requirement in radiology training, yet they are time-consuming to complete and prone to inconsistency when authored manually. This study investigates whether large language models (LLMs) can automate procedural case log documentation directly from free-text radiology reports. We evaluate multiple local and commercial LLMs under instruction-based and chain-of-thought prompting to extract structured procedural information from 414 curated interventional radiology reports authored by nine residents between 2018 and 2024. Model performance is assessed using sensitivity, specificity, and F1-score, alongside inference latency and token efficiency to estimate operational cost. Results show that both local and commercial models achieve strong extraction performance, with best F1-scores approaching 0.87, while exhibiting different trade-offs between speed and cost. Automation using LLMs has the potential to substantially reduce clerical burden for trainees and improve consistency in case logging. These findings demonstrate the feasibility of AI-assisted documentation in medical education and highlight the need for further validation across institutions and clinical workflows.