SearchLLM: Detecting LLM Paraphrased Text by Measuring the Similarity with Regeneration of the Candidate Source via Search Engine

TL;DR

SearchLLM tackles the difficulty of detecting LLM-paraphrased text by leveraging search-engine retrieved original sources and regenerating text from those sources to compare with the input. As a proxy, it can enhance existing detectors, improving accuracy on near-paraphrase cases and increasing robustness to paraphrasing attacks. Evaluations across RAID, MAGE, and XSum demonstrate consistent improvements in ROC AUC across detectors and models, including unknown LLMs/prompts. The approach highlights practical trade-offs, such as reliance on public sources and web-search costs, while offering a scalable, retrieval-augmented framework for LLM detection in real-world settings.

Abstract

With the advent of large language models (LLMs), it has become common practice for users to draft text and utilize LLMs to enhance its quality through paraphrasing. However, this process can sometimes result in the loss or distortion of the original intended meaning. Due to the human-like quality of LLM-generated text, traditional detection methods often fail, particularly when text is paraphrased to closely mimic original content. In response to these challenges, we propose a novel approach named SearchLLM, designed to identify LLM-paraphrased text by leveraging search engine capabilities to locate potential original text sources. By analyzing similarities between the input and regenerated versions of candidate sources, SearchLLM effectively distinguishes LLM-paraphrased content. SearchLLM is designed as a proxy layer, allowing seamless integration with existing detectors to enhance their performance. Experimental results across various LLMs demonstrate that SearchLLM consistently enhances the accuracy of recent detectors in detecting LLM-paraphrased text that closely mimics original content. Furthermore, SearchLLM also helps the detectors prevent paraphrasing attacks.

Figures (14)

• Figure 1: Illustration of variance in LLM-generated texts derived from a human text $h$. The comparison demonstrates minimal variance between the generated text $m$ and the regenerated text $r$.
• Figure 2: Comparison of similarity between samples produced by humans and LLMs with internet sources.
• Figure 3: Change in similarity between regenerated samples and human or LLM-generated samples.
• Figure 4: Overview of the $\mathrm{SearchLLM}$ schema for determining whether input text $t$ is human- or LLM-generated. The system considers three main cases: (1) $\mathrm{SearchLLM}$ compares $t$ with a candidate $t_c$ retrieved by a search engine to identify human text; (2) $\mathrm{SearchLLM}$ generates a regeneration $t_r$ from $t_c$ to determine LLM-generated text; (3) if neither case applies, $\mathrm{SearchLLM}$ delegates the decision to an existing method.
• Figure 5: Process of matching between input data and website content or re-generated text.