Detecting Spelling and Grammatical Anomalies in Russian Poetry Texts

TL;DR

The paper tackles the problem of detecting spelling and grammatical anomalies in Russian poetry to improve training data quality for generative models. It introduces the RUPOR dataset and the synthetic_GED corpus, and conducts a comprehensive evaluation showing perplexity and unsupervised outlier methods are inadequate for poetry, while supervised binary classifiers trained on synthetic distortions perform best. The findings demonstrate that large Russian-pretrained models yield superior performance over multilingual counterparts, highlighting the value of domain-specific data and supervision for grammaticality and defect detection in creative text. This work provides practical tools and datasets for data engineers and researchers to filter or select high-quality poetry data for training and evaluating generative models in creative domains. Overall, the study advances robust data-cleaning practices for Russian-language creative NLP tasks and emphasizes the importance of domain-aware evaluation in the poetry domain.

Abstract

The quality of natural language texts in fine-tuning datasets plays a critical role in the performance of generative models, particularly in computational creativity tasks such as poem or song lyric generation. Fluency defects in generated poems significantly reduce their value. However, training texts are often sourced from internet-based platforms without stringent quality control, posing a challenge for data engineers to manage defect levels effectively. To address this issue, we propose the use of automated linguistic anomaly detection to identify and filter out low-quality texts from training datasets for creative models. In this paper, we present a comprehensive comparison of unsupervised and supervised text anomaly detection approaches, utilizing both synthetic and human-labeled datasets. We also introduce the RUPOR dataset, a collection of Russian-language human-labeled poems designed for cross-sentence grammatical error detection, and provide the full evaluation code. Our work aims to empower the community with tools and insights to improve the quality of training datasets for generative models in creative domains.

Figures (5)

• Figure 1: Relationship between perplexity and token sequence length, illustrating how perplexity varies as the sequence length increases.
• Figure 2: Surprisal gaps for GMMs trained on ruRoberta-large embeddings and Russian-language text pairs.
• Figure 3: Grid search of $p_t$ threshold and corresponding $F_{0.5}$ metric on the RUPOR poetry.
• Figure 4: Grid search of $\xi$ threshold and corresponding $F_{0.5}$ metric on the RUPOR poetry.
• Figure 5: Permutation feature importance at the text level for a binary classifier in the linguistic anomaly detection task.