WikiSQE: A Large-Scale Dataset for Sentence Quality Estimation in Wikipedia

TL;DR

WikiSQE presents a large-scale, sentence-level quality estimation dataset for Wikipedia by extracting ~3.4 million sentences from the full English edit history and labeling them with 153 inline-template-derived quality cues, organized into five categories. The dataset construction combines source-text extraction, careful label selection, and a robust sentence extraction pipeline, enabling fine-grained analysis of citation, syntactic/semantic revision, information addition, and other quality aspects. Automated classifiers (e.g., DeBERTa, BERT, RoBERTa) achieve F1 scores of roughly 70–85%, with information addition being easier to detect and citation/semantic-revision/disputed-claim being more challenging; non-expert annotation generally trails behind the model except for a few labels. The work demonstrates the dataset’s potential for broad NLP tasks, discusses limitations such as data distribution and multilingual expansion, and points to future applications and extensions in quality control for large-scale text corpora.

Abstract

Wikipedia can be edited by anyone and thus contains various quality sentences. Therefore, Wikipedia includes some poor-quality edits, which are often marked up by other editors. While editors' reviews enhance the credibility of Wikipedia, it is hard to check all edited text. Assisting in this process is very important, but a large and comprehensive dataset for studying it does not currently exist. Here, we propose WikiSQE, the first large-scale dataset for sentence quality estimation in Wikipedia. Each sentence is extracted from the entire revision history of English Wikipedia, and the target quality labels were carefully investigated and selected. WikiSQE has about 3.4 M sentences with 153 quality labels. In the experiment with automatic classification using competitive machine learning models, sentences that had problems with citation, syntax/semantics, or propositions were found to be more difficult to detect. In addition, by performing human annotation, we found that the model we developed performed better than the crowdsourced workers. WikiSQE is expected to be a valuable resource for other tasks in NLP.