Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Identifying Reliable Evaluation Metrics for Scientific Text Revision

Léane Jourdan, Florian Boudin, Richard Dufour, Nicolas Hernandez

TL;DR

Evaluating revisions of scientific text is challenging because traditional similarity metrics do not capture meaningful improvements. The authors combine manual annotation, reference-free domain metrics, and LLM-based judges to identify reliable evaluation strategies, using the ParaRev dataset for empirical analysis. They find that LLM judges excel at following revision instructions but struggle with correctness, while domain-specific metrics provide complementary signals; a hybrid approach yields the most reliable assessment of revision quality. The work guides scalable, human-aligned evaluation for scientific writing revisions and informs the design of future automatic evaluation frameworks.

Abstract

Evaluating text revision in scientific writing remains a challenge, as traditional metrics such as ROUGE and BERTScore primarily focus on similarity rather than capturing meaningful improvements. In this work, we analyse and identify the limitations of these metrics and explore alternative evaluation methods that better align with human judgments. We first conduct a manual annotation study to assess the quality of different revisions. Then, we investigate reference-free evaluation metrics from related NLP domains. Additionally, we examine LLM-as-a-judge approaches, analysing their ability to assess revisions with and without a gold reference. Our results show that LLMs effectively assess instruction-following but struggle with correctness, while domain-specific metrics provide complementary insights. We find that a hybrid approach combining LLM-as-a-judge evaluation and task-specific metrics offers the most reliable assessment of revision quality.

Identifying Reliable Evaluation Metrics for Scientific Text Revision

TL;DR

Evaluating revisions of scientific text is challenging because traditional similarity metrics do not capture meaningful improvements. The authors combine manual annotation, reference-free domain metrics, and LLM-based judges to identify reliable evaluation strategies, using the ParaRev dataset for empirical analysis. They find that LLM judges excel at following revision instructions but struggle with correctness, while domain-specific metrics provide complementary signals; a hybrid approach yields the most reliable assessment of revision quality. The work guides scalable, human-aligned evaluation for scientific writing revisions and informs the design of future automatic evaluation frameworks.

Abstract

Evaluating text revision in scientific writing remains a challenge, as traditional metrics such as ROUGE and BERTScore primarily focus on similarity rather than capturing meaningful improvements. In this work, we analyse and identify the limitations of these metrics and explore alternative evaluation methods that better align with human judgments. We first conduct a manual annotation study to assess the quality of different revisions. Then, we investigate reference-free evaluation metrics from related NLP domains. Additionally, we examine LLM-as-a-judge approaches, analysing their ability to assess revisions with and without a gold reference. Our results show that LLMs effectively assess instruction-following but struggle with correctness, while domain-specific metrics provide complementary insights. We find that a hybrid approach combining LLM-as-a-judge evaluation and task-specific metrics offers the most reliable assessment of revision quality.

Paper Structure

This paper contains 38 sections, 17 figures, 8 tables.

Table of Contents

  1. Introduction
  2. État de l'art
  3. Métriques de similarité des n-grammes
  4. Métriques de similarité des plongements
  5. Approches GML-Juge
  6. Protocole expérimental
  7. Jeu de données
  8. Modèles de révision
  9. Tâche d'annotation
  10. Résultats de la phase d'annotation
  11. Limites des métriques basées sur la similarité
  12. Performances des modèles de révision avec les métriques basées sur la similarité
  13. Redondance et corrélation entre métriques
  14. Sensibilité des métriques à la distance d'édition
  15. Exploration d'approches alternatives pour l'évaluation
  16. ...and 23 more sections

Figures (17)

  • Figure 1: Aperçu de la tâche de révision
  • Figure 2: Distribution de la préférence étendue humaine pour chaque modèle de révision. La zone verte correspond aux cas où le modèle est préféré.
  • Figure 3: Corrélation des métriques de similarité
  • Figure 4: Corrélation des métriques connexes
  • Figure 5: Alignement des métriques automatiques avec les annotations humaines par catégorie de révision
  • ...and 12 more figures