Large Language Models can be Guided to Evade AI-Generated Text Detection

TL;DR

This work shows that large language models can be steered via carefully engineered prompts to evade AI-detection systems, challenging the robustness of existing detectors. It introduces Substitution-based In-Context example Optimization (SICO), a prompt-construction framework guided by a proxy detector that iteratively substitutes words and phrases in demonstrations to produce human-like outputs. Across three real-world tasks, SICO significantly lowers detectors' AUC (by about $0.5$ on average) and yields text with readability and task completion rates comparable to humans, while remaining cost-efficient. The paper also discusses defense strategies, including training detectors on SICO data and ensemble approaches, and frames the evolution of AI detection as an arms race that requires ongoing robustness research.

Abstract

Large language models (LLMs) have shown remarkable performance in various tasks and have been extensively utilized by the public. However, the increasing concerns regarding the misuse of LLMs, such as plagiarism and spamming, have led to the development of multiple detectors, including fine-tuned classifiers and statistical methods. In this study, we equip LLMs with prompts, rather than relying on an external paraphraser, to evaluate the vulnerability of these detectors. We propose a novel Substitution-based In-Context example Optimization method (SICO) to automatically construct prompts for evading the detectors. SICO is cost-efficient as it requires only 40 human-written examples and a limited number of LLM inferences to generate a prompt. Moreover, once a task-specific prompt has been constructed, it can be universally used against a wide range of detectors. Extensive experiments across three real-world tasks demonstrate that SICO significantly outperforms the paraphraser baselines and enables GPT-3.5 to successfully evade six detectors, decreasing their AUC by 0.5 on average. Furthermore, a comprehensive human evaluation show that the SICO-generated text achieves human-level readability and task completion rates, while preserving high imperceptibility. Finally, we propose an ensemble approach to enhance the robustness of detectors against SICO attack. The code is publicly available at https://github.com/ColinLu50/Evade-GPT-Detector.

Figures (6)

• Figure 1: Illustration of how SICO generates prompts for the question answering task. The probability $P_{\text{AI}}$, as predicted by the proxy detector, indicates the likelihood that the given text is AI-generated. Once SICO prompt is constructed, it serves as a template, allowing users to insert various task inputs (highlighted in purple text).
• Figure 2: ROC curves for six detectors evaluating text generated by various evasion methods in an academic writing task.
• Figure 3: The trajectory of the $\mathcal{U}(p^*)$ during prompt optimization. This plot is derived from three distinct training runs on three tasks.
• Figure 4: ROC curves.
• Figure 5: The interface of the annotation platform used in our experiment.