Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparative Experimentation of Accuracy Metrics in Automated Medical Reporting: The Case of Otitis Consultations

Wouter Faber, Renske Eline Bootsma, Tom Huibers, Sandra van Dulmen, Sjaak Brinkkemper

TL;DR

This work evaluates the accuracy of AI-generated medical reports for Otitis consultations by comparing 10 metrics against GP-authored references. It introduces a Composite Accuracy Score (CAS) to synthesize metric performance and uses Care2Report data with GPT-4.0 to generate Dutch SOAP/SOEP-style reports, assessing correlations with human-evaluated omissions, inaccuracies, and added content. The study finds ROUGE-L and Word Mover's Distance (WMD) as the most favorable metrics under CAS and post-edit-time considerations, challenging conclusions from prior work and highlighting domain-specific effects. The results offer guidance for selecting metrics in automated medical reporting and underscore the practical impact on reducing clinicians’ administrative workload, while noting limitations due to data size and language specifics.

Abstract

Generative Artificial Intelligence (AI) can be used to automatically generate medical reports based on transcripts of medical consultations. The aim is to reduce the administrative burden that healthcare professionals face. The accuracy of the generated reports needs to be established to ensure their correctness and usefulness. There are several metrics for measuring the accuracy of AI generated reports, but little work has been done towards the application of these metrics in medical reporting. A comparative experimentation of 10 accuracy metrics has been performed on AI generated medical reports against their corresponding General Practitioner's (GP) medical reports concerning Otitis consultations. The number of missing, incorrect, and additional statements of the generated reports have been correlated with the metric scores. In addition, we introduce and define a Composite Accuracy Score which produces a single score for comparing the metrics within the field of automated medical reporting. Findings show that based on the correlation study and the Composite Accuracy Score, the ROUGE-L and Word Mover's Distance metrics are the preferred metrics, which is not in line with previous work. These findings help determine the accuracy of an AI generated medical report, which aids the development of systems that generate medical reports for GPs to reduce the administrative burden.

Comparative Experimentation of Accuracy Metrics in Automated Medical Reporting: The Case of Otitis Consultations

TL;DR

This work evaluates the accuracy of AI-generated medical reports for Otitis consultations by comparing 10 metrics against GP-authored references. It introduces a Composite Accuracy Score (CAS) to synthesize metric performance and uses Care2Report data with GPT-4.0 to generate Dutch SOAP/SOEP-style reports, assessing correlations with human-evaluated omissions, inaccuracies, and added content. The study finds ROUGE-L and Word Mover's Distance (WMD) as the most favorable metrics under CAS and post-edit-time considerations, challenging conclusions from prior work and highlighting domain-specific effects. The results offer guidance for selecting metrics in automated medical reporting and underscore the practical impact on reducing clinicians’ administrative workload, while noting limitations due to data size and language specifics.

Abstract

Generative Artificial Intelligence (AI) can be used to automatically generate medical reports based on transcripts of medical consultations. The aim is to reduce the administrative burden that healthcare professionals face. The accuracy of the generated reports needs to be established to ensure their correctness and usefulness. There are several metrics for measuring the accuracy of AI generated reports, but little work has been done towards the application of these metrics in medical reporting. A comparative experimentation of 10 accuracy metrics has been performed on AI generated medical reports against their corresponding General Practitioner's (GP) medical reports concerning Otitis consultations. The number of missing, incorrect, and additional statements of the generated reports have been correlated with the metric scores. In addition, we introduce and define a Composite Accuracy Score which produces a single score for comparing the metrics within the field of automated medical reporting. Findings show that based on the correlation study and the Composite Accuracy Score, the ROUGE-L and Word Mover's Distance metrics are the preferred metrics, which is not in line with previous work. These findings help determine the accuracy of an AI generated medical report, which aids the development of systems that generate medical reports for GPs to reduce the administrative burden.
Paper Structure (25 sections, 1 equation, 1 figure, 4 tables)

This paper contains 25 sections, 1 equation, 1 figure, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Accuracy Metrics
  4. Comparison of Metrics
  5. SOEP Reporting for GPs
  6. RESEARCH METHOD
  7. Materials
  8. Pre-study
  9. Prompt for Report Generation
  10. Metric Selection and Execution
  11. Human evaluation
  12. FINDINGS
  13. Human Evaluation
  14. Correlation between Metrics and Human Evaluation Aspects
  15. DISCUSSION
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Research Method Diagram, showing the method along with its input and intended output.