Evaluating local large language models for structured extraction from endometriosis-specific transvaginal ultrasound reports

TL;DR

This work addresses the challenge of converting unstructured endometriosis transvaginal ultrasound reports into structured data using locally deployed LLMs to support privacy-preserving imaging informatics. It benchmarks three on-premise models against expert extraction on 49 reports, revealing that a 20B parameter model achieves the best balance of accuracy (86.02%) and deployability, while smaller models underperform on clinical semantics. A highly complementary error profile emerges: the LLM excels at syntactic and formatting tasks, whereas humans outperform in semantic interpretation, underscoring the value of a human-in-the-loop workflow for high-stakes clinical data extraction. The findings advocate for HITL systems where LLMs automate routine structuring and flag potential human errors, enabling clinicians to validate semantically complex fields and improve overall data quality in structured reporting.

Abstract

In this study, we evaluate a locally-deployed large-language model (LLM) to convert unstructured endometriosis transvaginal ultrasound (eTVUS) scan reports into structured data for imaging informatics workflows. Across 49 eTVUS reports, we compared three LLMs (7B/8B and a 20B-parameter model) against expert human extraction. The 20B model achieved a mean accuracy of 86.02%, substantially outperforming smaller models and confirming the importance of scale in handling complex clinical text. Crucially, we identified a highly complementary error profile: the LLM excelled at syntactic consistency (e.g., date/numeric formatting) where humans faltered, while human experts provided superior semantic and contextual interpretation. We also found that the LLM's semantic errors were fundamental limitations that could not be mitigated by simple prompt engineering. These findings strongly support a human-in-the-loop (HITL) workflow in which the on-premise LLM serves as a collaborative tool, not a full replacement. It automates routine structuring and flags potential human errors, enabling imaging specialists to focus on high-level semantic validation. We discuss implications for structured reporting and interactive AI systems in clinical practice.

Figures (1)

• Figure 1: Performance breakdown by data field type. The LLM excels at structured fields (Date, Numeric).