DocLens: Multi-aspect Fine-grained Evaluation for Medical Text Generation

TL;DR

A set of metrics to evaluate the completeness, conciseness, and attribution of the generated text at a fine-grained level are proposed and demonstrated and the effectiveness of the resulting framework, DocLens, is demonstrated.

Abstract

Medical text generation aims to assist with administrative work and highlight salient information to support decision-making. To reflect the specific requirements of medical text, in this paper, we propose a set of metrics to evaluate the completeness, conciseness, and attribution of the generated text at a fine-grained level. The metrics can be computed by various types of evaluators including instruction-following (both proprietary and open-source) and supervised entailment models. We demonstrate the effectiveness of the resulting framework, DocLens, with three evaluators on three tasks: clinical note generation, radiology report summarization, and patient question summarization. A comprehensive human study shows that DocLens exhibits substantially higher agreement with the judgments of medical experts than existing metrics. The results also highlight the need to improve open-source evaluators and suggest potential directions.

Figures (7)

• Figure 1: Evaluation aspects of DocLens for medical text generation. Completeness evaluates the amount of salient details in the system output. Conciseness evaluates the amount of information that is both accurate and salient. Attribution checks whether the generated information can be traced back and attributed from the input
• Figure 2: To conduct a multi-aspect evaluation, we verify the entailment relations among the input (e.g., patient-doctor dialogue), system output (e.g., generated clinical note), and reference (e.g., human-written clinical note).
• Figure 3: Illustration of the metrics of DocLens for medical evaluation: Claim Recall measures the proportion of claims from the human-written reference that can be entailed by the system output. Claim Precision measures the proportion of claims from the output that can be entailed by the reference. Citation Recall measures the proportion of output statements that can be entailed by their corresponding citations. Citation Precision measures the proportion of citations that factually support the associated statement.
• Figure 4: Clinical note generation results on ACI-BENCH aci-bench. We split the results under existing metrics and DocLens computed with GPT-4. We evaluate open-source and proprietary note generation models with different numbers of in-context examples.
• Figure 7: Agreement between each metric and the subjective preferences of medical experts over two system outputs. We only annotate the system outputs pairs where the two metrics have different preferences. The outputs are selected from the Objective Exam (O-Exam) and Assessment and Plan (A&P) section in note generation.