Humanizing the Machine: Proxy Attacks to Mislead LLM Detectors
Tianchun Wang, Yuanzhou Chen, Zichuan Liu, Zhanwen Chen, Haifeng Chen, Xiang Zhang, Wei Cheng
TL;DR
The paper addresses the vulnerability of zero-shot LLM detectors to decoding-time evasion by introducing HUMPA, a proxy attack that uses a fine-tuned small, humanized language model during decoding to bias large LLM outputs toward human-like text without touching the large model itself. The approach blends preference-based RL and Direct Preference Optimization to train a small SLM, then applies a logit-offset at inference with an attack ratio $\alpha$ to emulate a human-like distribution $\pi_H$, showing that this can be equivalently viewed as fine-tuning the large model with $\beta_0/\alpha$. Comprehensive experiments across multiple open-source LLMs, detectors, and across-domain and cross-language settings demonstrate substantial AUROC degradation (average around $70.4\%$, up to $95.0\%$) while maintaining text quality within a modest budget. The findings reveal notable detector vulnerabilities and underscore the need for robust defenses that account for decoding-time distribution shifts, with practical implications for policy and security in AI deployment.
Abstract
The advent of large language models (LLMs) has revolutionized the field of text generation, producing outputs that closely mimic human-like writing. Although academic and industrial institutions have developed detectors to prevent the malicious usage of LLM-generated texts, other research has doubt about the robustness of these systems. To stress test these detectors, we introduce a proxy-attack strategy that effortlessly compromises LLMs, causing them to produce outputs that align with human-written text and mislead detection systems. Our method attacks the source model by leveraging a reinforcement learning (RL) fine-tuned humanized small language model (SLM) in the decoding phase. Through an in-depth analysis, we demonstrate that our attack strategy is capable of generating responses that are indistinguishable to detectors, preventing them from differentiating between machine-generated and human-written text. We conduct systematic evaluations on extensive datasets using proxy-attacked open-source models, including Llama2-13B, Llama3-70B, and Mixtral-8*7B in both white- and black-box settings. Our findings show that the proxy-attack strategy effectively deceives the leading detectors, resulting in an average AUROC drop of 70.4% across multiple datasets, with a maximum drop of 90.3% on a single dataset. Furthermore, in cross-discipline scenarios, our strategy also bypasses these detectors, leading to a significant relative decrease of up to 90.9%, while in cross-language scenario, the drop reaches 91.3%. Despite our proxy-attack strategy successfully bypassing the detectors with such significant relative drops, we find that the generation quality of the attacked models remains preserved, even within a modest utility budget, when compared to the text produced by the original, unattacked source model.