BUSTED at AraGenEval Shared Task: A Comparative Study of Transformer-Based Models for Arabic AI-Generated Text Detection

TL;DR

This work tackles Arabic AI-generated text detection by comparing three transformer-based approaches (AraELECTRA, CAMeLBERT, XLM-RoBERTa) on the AraGenEval shared task. Fine-tuning for binary classification, the multilingual XLM-RoBERTa model delivers the best performance with a macro F1 of $0.7701$, outperforming Arabic-focused baselines and illustrating the generalization benefits of multilingual pretraining. The study also reveals that aggressive text normalization can degrade signals essential for distinguishing human from machine text, underscoring the nuanced role of preprocessing. The results highlight the potential of multilingual models in low-resource Arabic NLP tasks and point to future work in ensembling and error-focused analyses to further improve detector robustness.

Abstract

This paper details our submission to the AraGenEval Shared Task on Arabic AI-generated text detection, where our team, BUSTED, secured 5th place. We investigated the effectiveness of three pre-trained transformer models: AraELECTRA, CAMeLBERT, and XLM-RoBERTa. Our approach involved fine-tuning each model on the provided dataset for a binary classification task. Our findings revealed a surprising result: the multilingual XLM-RoBERTa model achieved the highest performance with an F1 score of 0.7701, outperforming the specialized Arabic models. This work underscores the complexities of AI-generated text detection and highlights the strong generalization capabilities of multilingual models.

Figures (2)

• Figure 1: Statistics of the AraGenEval training dataset. The classes are well-balanced, but human-written texts are more than twice as long as machine-generated ones.
• Figure 2: Overview of our comparative system. Input text is processed in parallel by three separate fine-tuned models. AraELECTRA's pipeline includes an additional text normalization step.