Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension

TL;DR

This work tackles whether automatic text simplification preserves the original meaning when readers extract information from simplified texts. It introduces a reading-comprehension–based human evaluation framework and applies it to compare a human-written simplification with nine automatic TS systems across paragraph-long passages. The findings show that supervised, pre-trained systems approach human performance in meaning preservation but still render at least 14% of questions unanswerable, while simpler models like KIS underperform substantially; SARI emerges as the most reliable automatic metric for system ranking in this setting. The study highlights the need for machine-in-the-loop validation and improved deletion control to ensure reliable meaning preservation in practical TS applications.

Abstract

Automatic text simplification (TS) aims to automate the process of rewriting text to make it easier for people to read. A pre-requisite for TS to be useful is that it should convey information that is consistent with the meaning of the original text. However, current TS evaluation protocols assess system outputs for simplicity and meaning preservation without regard for the document context in which output sentences occur and for how people understand them. In this work, we introduce a human evaluation framework to assess whether simplified texts preserve meaning using reading comprehension questions. With this framework, we conduct a thorough human evaluation of texts by humans and by nine automatic systems. Supervised systems that leverage pre-training knowledge achieve the highest scores on the reading comprehension (RC) tasks amongst the automatic controllable TS systems. However, even the best-performing supervised system struggles with at least 14% of the questions, marking them as "unanswerable'' based on simplified content. We further investigate how existing TS evaluation metrics and automatic question-answering systems approximate the human judgments we obtained.

Figures (6)

• Figure 1: RC questions to be answered after reading either the original or the simplified text. The answer options include the correct answer (A), three incorrect options of varying difficulty (B,C,D), and an option (E) that captures questions rendered unanswerable after automatic TS.
• Figure 2: The accuracy scores per condition are averaged over $50$ runs for each subset-size $k$: Rankings stabilize with a sample size of 40 passages.
• Figure 3: Number of questions marked with UA per paragraph: Different systems have different passage-questions pairs marked as unanswerable, suggesting different deletion errors.
• Figure 4: Unigram Overlap between the answer options and the passage (Support (A)), question and the passage (Support(Q)) and product of the two (Support (A)*Support (Q)) indicates content deletion as a major factor in making a question unanswerable. TPR: True Positive Rate; FPR: False Positive Rate.
• Figure 5: Participants mark 14-65% questions with UA, suggesting that the meaning of the original text is not entirely preserved in the simplified texts.