M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection

TL;DR

M4GT-Bench introduces a comprehensive, multilingual, multi-domain benchmark for detecting machine-generated text across three tasks: binary classification, generator attribution, and boundary detection in mixed human-machine text. It assembles a diverse corpus (nine languages, six domains, nine generators) and provides datasets, cleaning procedures, and evaluation metrics for each task, enabling thorough cross-domain and cross-generator evaluation. Across baselines (RoBERTa, XLM-R, GLTR, Stylistic, NELA) and feature-based approaches, results reveal substantial generalization gaps when detectors encounter unseen domains or generators, with XLM-R typically offering the strongest overall performance and boundary detection proving particularly challenging. The work also includes human evaluation, demonstrating that humans struggle to distinguish among generators, underscoring the need for robust, domain-aware detectors and inspiring future extensions to multimedia modalities and more complex mixed-text scenarios.

Abstract

The advent of Large Language Models (LLMs) has brought an unprecedented surge in machine-generated text (MGT) across diverse channels. This raises legitimate concerns about its potential misuse and societal implications. The need to identify and differentiate such content from genuine human-generated text is critical in combating disinformation, preserving the integrity of education and scientific fields, and maintaining trust in communication. In this work, we address this problem by introducing a new benchmark based on a multilingual, multi-domain, and multi-generator corpus of MGTs -- M4GT-Bench. The benchmark is compiled of three tasks: (1) mono-lingual and multi-lingual binary MGT detection; (2) multi-way detection where one need to identify, which particular model generated the text; and (3) mixed human-machine text detection, where a word boundary delimiting MGT from human-written content should be determined. On the developed benchmark, we have tested several MGT detection baselines and also conducted an evaluation of human performance. We see that obtaining good performance in MGT detection usually requires an access to the training data from the same domain and generators. The benchmark is available at https://github.com/mbzuai-nlp/M4GT-Bench.

Figures (2)

• Figure 1: We split 140 examples into four groups, each involving three domains and four generators, with 48 examples including five demonstrations for learning.
• Figure 2: Task 3 prompt templates used to generate continuations of paper reviews and student essays.