Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CASIMIR: A Corpus of Scientific Articles enhanced with Multiple Author-Integrated Revisions

Leane Jourdan, Florian Boudin, Nicolas Hernandez, Richard Dufour

TL;DR

CASIMIR addresses the need for large-scale resources to study and improve revision in scientific writing by providing the largest corpus of multi-version, peer-reviewed articles from OpenReview, with sentence-level alignment, paragraph localization, and automatic revision-intention labels. The authors construct 36,733 version-pairs across 15,646 English-language articles, annotated with 5.2 million edits, and paired with associated peer reviews and venue metadata. They perform a thorough corpus analysis and benchmark multiple open-source revision models (IteraTeR-PEGASUS, CoEdIT, Llama2-7B) using EM, SARI, BLEU, ROUGE-L, and Bertscore, highlighting that traditional metrics may not fully capture revision quality and that a simple CopyInput baseline can be competitive. The work demonstrates the dataset’s potential to train discourse-aware revision tools, study revision dynamics, and support tasks like review generation and acceptance prediction, while also calling for improved evaluation methods and careful handling of conversion and labeling noise.

Abstract

Writing a scientific article is a challenging task as it is a highly codified and specific genre, consequently proficiency in written communication is essential for effectively conveying research findings and ideas. In this article, we propose an original textual resource on the revision step of the writing process of scientific articles. This new dataset, called CASIMIR, contains the multiple revised versions of 15,646 scientific articles from OpenReview, along with their peer reviews. Pairs of consecutive versions of an article are aligned at sentence-level while keeping paragraph location information as metadata for supporting future revision studies at the discourse level. Each pair of revised sentences is enriched with automatically extracted edits and associated revision intention. To assess the initial quality on the dataset, we conducted a qualitative study of several state-of-the-art text revision approaches and compared various evaluation metrics. Our experiments led us to question the relevance of the current evaluation methods for the text revision task.

CASIMIR: A Corpus of Scientific Articles enhanced with Multiple Author-Integrated Revisions

TL;DR

CASIMIR addresses the need for large-scale resources to study and improve revision in scientific writing by providing the largest corpus of multi-version, peer-reviewed articles from OpenReview, with sentence-level alignment, paragraph localization, and automatic revision-intention labels. The authors construct 36,733 version-pairs across 15,646 English-language articles, annotated with 5.2 million edits, and paired with associated peer reviews and venue metadata. They perform a thorough corpus analysis and benchmark multiple open-source revision models (IteraTeR-PEGASUS, CoEdIT, Llama2-7B) using EM, SARI, BLEU, ROUGE-L, and Bertscore, highlighting that traditional metrics may not fully capture revision quality and that a simple CopyInput baseline can be competitive. The work demonstrates the dataset’s potential to train discourse-aware revision tools, study revision dynamics, and support tasks like review generation and acceptance prediction, while also calling for improved evaluation methods and careful handling of conversion and labeling noise.

Abstract

Writing a scientific article is a challenging task as it is a highly codified and specific genre, consequently proficiency in written communication is essential for effectively conveying research findings and ideas. In this article, we propose an original textual resource on the revision step of the writing process of scientific articles. This new dataset, called CASIMIR, contains the multiple revised versions of 15,646 scientific articles from OpenReview, along with their peer reviews. Pairs of consecutive versions of an article are aligned at sentence-level while keeping paragraph location information as metadata for supporting future revision studies at the discourse level. Each pair of revised sentences is enriched with automatically extracted edits and associated revision intention. To assess the initial quality on the dataset, we conducted a qualitative study of several state-of-the-art text revision approaches and compared various evaluation metrics. Our experiments led us to question the relevance of the current evaluation methods for the text revision task.
Paper Structure (30 sections, 5 figures, 3 tables)

This paper contains 30 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Scientific writing process
  4. Revision datasets in the scientific domain
  5. Corpus Creation
  6. Large Data Collection
  7. PDF-to-Text Conversion
  8. Alignment and Edit Extraction
  9. Edit Types Labelling
  10. Corpus split
  11. Corpus Analysis
  12. Distribution of versions and reviews
  13. Distribution of edits
  14. Evolution and location of edits
  15. Experiments with Text Revision Models
  16. ...and 15 more sections

Figures (5)

  • Figure 1: The writing process of a scientific article, inspired by seow2002writing
  • Figure 2: Steps of the creation process of the CASIMIR corpus
  • Figure 3: Distribution of articles by number of versions (left), edits (center) and reviews (right)
  • Figure 4: Evolution of edited text percentage in articles by revision depth.
  • Figure 5: Evolution of the location of edited text by intention and revision depth