TH-Bench: Evaluating Evading Attacks via Humanizing AI Text on Machine-Generated Text Detectors

TL;DR

This work tackles the lack of a unified framework for evaluating evading attacks on machine-generated text detectors. It introduces TH-Bench, a three-dimensional benchmark (attack effectiveness, text quality, and execution overhead) to assess six attacks across 13 detectors, six datasets, and 11 LLMs. The study reveals no single attack excels in all dimensions, uncovers a fundamental trade-off among the three metrics, and proposes two optimization strategies—Quality-Preserving Attack (QPA) and Attack Blending—with preliminary validation. These insights offer guidance for detector robustness, risk assessment, and future research into more holistic evasion strategies and defenses.

Abstract

As Large Language Models (LLMs) advance, Machine-Generated Texts (MGTs) have become increasingly fluent, high-quality, and informative. Existing wide-range MGT detectors are designed to identify MGTs to prevent the spread of plagiarism and misinformation. However, adversaries attempt to humanize MGTs to evade detection (named evading attacks), which requires only minor modifications to bypass MGT detectors. Unfortunately, existing attacks generally lack a unified and comprehensive evaluation framework, as they are assessed using different experimental settings, model architectures, and datasets. To fill this gap, we introduce the Text-Humanization Benchmark (TH-Bench), the first comprehensive benchmark to evaluate evading attacks against MGT detectors. TH-Bench evaluates attacks across three key dimensions: evading effectiveness, text quality, and computational overhead. Our extensive experiments evaluate 6 state-of-the-art attacks against 13 MGT detectors across 6 datasets, spanning 19 domains and generated by 11 widely used LLMs. Our findings reveal that no single evading attack excels across all three dimensions. Through in-depth analysis, we highlight the strengths and limitations of different attacks. More importantly, we identify a trade-off among three dimensions and propose two optimization insights. Through preliminary experiments, we validate their correctness and effectiveness, offering potential directions for future research.

Figures (15)

• Figure 1: TH-Bench benchmarks evading attacks in terms of effectiveness against MGT detectors, impact on text quality, and computational overhead. The right figure shows the normalized results of various attacks we evaluate across three dimensions. A higher value for each point indicates better performance of the attack in the corresponding metric.
• Figure 2: The AUC performance of evading attacks across LLMs, averaged across detectors.
• Figure 3: The AUC performance of Task attribution task for evading attacks against LM-D detector, averaged across LLMs.
• Figure 4: The AUC value that evaluates attack effectiveness in a real-world scenario.
• Figure 5: Text quality comparison of different LLMs: left panel for MGTBench, right for MGT-academic, with values averaged across all fields within each dataset.