Zero-shot Cross-Lingual Transfer for Synthetic Data Generation in Grammatical Error Detection

TL;DR

This work tackles grammatical error detection in low-resource languages by leveraging zero-shot cross-lingual transfer with multilingual pre-trained language models. It introduces a two-stage fine-tuning pipeline that first trains on multilingual synthetic data generated by a language-agnostic back-translation error generator and then fine-tunes on human-annotated source-language data, achieving state-of-the-art results among annotation-free GED methods across multiple target languages. A key finding is that the synthetic data, when produced via cross-lingual generative modeling, yields more diverse and human-like errors than prior approaches, and the approach scales with additional source languages. The authors release a synthetic GED corpus exceeding $5$ million samples across $11$ languages, demonstrating practical impact for expanding GED coverage in low-resource contexts. Overall, the method advances annotation-free GED and suggests broader applicability of cross-lingual transfer for data generation in multilingual NLP tasks.

Abstract

Grammatical Error Detection (GED) methods rely heavily on human annotated error corpora. However, these annotations are unavailable in many low-resource languages. In this paper, we investigate GED in this context. Leveraging the zero-shot cross-lingual transfer capabilities of multilingual pre-trained language models, we train a model using data from a diverse set of languages to generate synthetic errors in other languages. These synthetic error corpora are then used to train a GED model. Specifically we propose a two-stage fine-tuning pipeline where the GED model is first fine-tuned on multilingual synthetic data from target languages followed by fine-tuning on human-annotated GED corpora from source languages. This approach outperforms current state-of-the-art annotation-free GED methods. We also analyse the errors produced by our method and other strong baselines, finding that our approach produces errors that are more diverse and more similar to human errors.

Figures (5)

• Figure 1: Overview of our proposed method.
• Figure 2: Precision-Recall curves comparing our method in different data configurations to our baselines.
• Figure 3: Relative improvement in terms of $F_{0.5}$ score compared to English-only fine-tuning as additional source languages are incorporated.
• Figure 4: Top 10 error types distribution of different annotation-free synthetic data generation methods.
• Figure 5: Normalized Entropy comparison of authentic and synthetic errors aggregated over different datasets.