Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation

Yue Guo, Jae Ho Sohn, Gondy Leroy, Trevor Cohen

TL;DR

This study probes whether LLM-generated plain language summaries (PLSs) truly aid lay understanding. It combines subjective quality ratings with objective comprehension tests in a large crowdsourced evaluation, contrasting six LLM-optimized PLS variants against human-written PLSs on 50 CELLS-derived pairs. The findings show a disconnect: humans achieve better comprehension despite LLMs matching perceived quality, and most automated metrics fail to predict understanding, though QA-based measures like QAEval and coherence-related scores show some promise. The work highlights the need for comprehension-focused generation and evaluation frameworks that prioritize factual content and background explanations to empower non-expert readers.

Abstract

Plain language summaries (PLSs) are essential for facilitating effective communication between clinicians and patients by making complex medical information easier for laypeople to understand and act upon. Large language models (LLMs) have recently shown promise in automating PLS generation, but their effectiveness in supporting health information comprehension remains unclear. Prior evaluations have generally relied on automated scores that do not measure understandability directly, or subjective Likert-scale ratings from convenience samples with limited generalizability. To address these gaps, we conducted a large-scale crowdsourced evaluation of LLM-generated PLSs using Amazon Mechanical Turk with 150 participants. We assessed PLS quality through subjective Likert-scale ratings focusing on simplicity, informativeness, coherence, and faithfulness; and objective multiple-choice comprehension and recall measures of reader understanding. Additionally, we examined the alignment between 10 automated evaluation metrics and human judgments. Our findings indicate that while LLMs can generate PLSs that appear indistinguishable from human-written ones in subjective evaluations, human-written PLSs lead to significantly better comprehension. Furthermore, automated evaluation metrics fail to reflect human judgment, calling into question their suitability for evaluating PLSs. This is the first study to systematically evaluate LLM-generated PLSs based on both reader preferences and comprehension outcomes. Our findings highlight the need for evaluation frameworks that move beyond surface-level quality and for generation methods that explicitly optimize for layperson comprehension.

Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation

TL;DR

This study probes whether LLM-generated plain language summaries (PLSs) truly aid lay understanding. It combines subjective quality ratings with objective comprehension tests in a large crowdsourced evaluation, contrasting six LLM-optimized PLS variants against human-written PLSs on 50 CELLS-derived pairs. The findings show a disconnect: humans achieve better comprehension despite LLMs matching perceived quality, and most automated metrics fail to predict understanding, though QA-based measures like QAEval and coherence-related scores show some promise. The work highlights the need for comprehension-focused generation and evaluation frameworks that prioritize factual content and background explanations to empower non-expert readers.

Abstract

Plain language summaries (PLSs) are essential for facilitating effective communication between clinicians and patients by making complex medical information easier for laypeople to understand and act upon. Large language models (LLMs) have recently shown promise in automating PLS generation, but their effectiveness in supporting health information comprehension remains unclear. Prior evaluations have generally relied on automated scores that do not measure understandability directly, or subjective Likert-scale ratings from convenience samples with limited generalizability. To address these gaps, we conducted a large-scale crowdsourced evaluation of LLM-generated PLSs using Amazon Mechanical Turk with 150 participants. We assessed PLS quality through subjective Likert-scale ratings focusing on simplicity, informativeness, coherence, and faithfulness; and objective multiple-choice comprehension and recall measures of reader understanding. Additionally, we examined the alignment between 10 automated evaluation metrics and human judgments. Our findings indicate that while LLMs can generate PLSs that appear indistinguishable from human-written ones in subjective evaluations, human-written PLSs lead to significantly better comprehension. Furthermore, automated evaluation metrics fail to reflect human judgment, calling into question their suitability for evaluating PLSs. This is the first study to systematically evaluate LLM-generated PLSs based on both reader preferences and comprehension outcomes. Our findings highlight the need for evaluation frameworks that move beyond surface-level quality and for generation methods that explicitly optimize for layperson comprehension.
Paper Structure (15 sections, 3 figures, 7 tables)

This paper contains 15 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Results
  3. Crowdsourced workers’ characteristics
  4. LLM- and human-generated PLSs' characteristics
  5. LLM- and human-generated PLSs have similar subjective ratings
  6. LLM-generated PLSs are harder to understand
  7. Subjective ratings predict comprehension
  8. Automated scores align poorly with human judgment
  9. Human factors in PLS evaluation
  10. Discussion & Conclusion
  11. Method
  12. Dataset
  13. Evaluation Procedure and Question Design
  14. Automated Metrics
  15. Analysis

Figures (3)

  • Figure 1: Comparison of linguistic and readability metrics across human-written and LLM-generated PLS under various optimization strategies (N = 1346). Metrics include paragraph length, vocabulary size, word familiarity, and three standard readability scores (Flesch-Kincaid, SMOG, Automated Readability Index). The red dashed line indicates the mean score for human-written PLS. Asterisks (*) denote statistically significant differences (paired t-test, p < 0.05) between each system and the human-written PLS. $p$-values are adjusted for multiple comparisons using the Benjamini--Hochberg correction.
  • Figure 2: Subjective and objective evaluation of human-written and LLM-generated PLS. The left panel ("Subjective Evaluation") shows mean human Likert-scale ratings across five quality dimensions: simplicity, informativeness, coherence, faithfulness, and providing necessary background knowledge (N = 1346). The right panel ("Objective Evaluation") presents performance on comprehension tasks: multiple-choice questions (MCQ) accuracy and recall rate. Colored bars represent subjective dimensions; hatched bars indicate MCQ and Recall scores, with distinct patterns. Asterisks (*) mark statistically significant differences to the human-written PLS baseline (paired t-test, p < 0.05). $p$-values are adjusted for multiple comparisons using the Benjamini--Hochberg correction.
  • Figure 3: Heatmaps of mean task time grouped by multiple-choice question (MCQ) accuracy (left) and recall rate (right), with rows representing self-identified familiarity (N = 1346). Color indicates average work time (in minutes). Each cell shows both the count of participants and the percentage within the corresponding familiarity group.