DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection

TL;DR

The paper tackles the challenge of detecting LLM-generated text in black-box settings by decoupling prompt information from intrinsic model characteristics. DPIC uses an auxiliary LLM to reconstruct a prompt for a given candidate text, regenerates text from that prompt, and then measures the similarity between the original and regenerated texts as an intrinsic feature, fed into a Siamese classifier. This approach improves generalization across domains and holds up against attacks, achieving substantial performance gains over baselines, including for advanced closed-source models like GPT-4 and Claude3. The work demonstrates practical viability with open auxiliary models and highlights DPIC's potential to enhance trust in AI-generated content safeguards.

Abstract

Large language models (LLMs) have the potential to generate texts that pose risks of misuse, such as plagiarism, planting fake reviews on e-commerce platforms, or creating inflammatory false tweets. Consequently, detecting whether a text is generated by LLMs has become increasingly important. Existing high-quality detection methods usually require access to the interior of the model to extract the intrinsic characteristics. However, since we do not have access to the interior of the black-box model, we must resort to surrogate models, which impacts detection quality. In order to achieve high-quality detection of black-box models, we would like to extract deep intrinsic characteristics of the black-box model generated texts. We view the generation process as a coupled process of prompt and intrinsic characteristics of the generative model. Based on this insight, we propose to decouple prompt and intrinsic characteristics (DPIC) for LLM-generated text detection method. Specifically, given a candidate text, DPIC employs an auxiliary LLM to reconstruct the prompt corresponding to the candidate text, then uses the prompt to regenerate text by the auxiliary LLM, which makes the candidate text and the regenerated text align with their prompts, respectively. Then, the similarity between the candidate text and the regenerated text is used as a detection feature, thus eliminating the prompt in the detection process, which allows the detector to focus on the intrinsic characteristics of the generative model. Compared to the baselines, DPIC has achieved an average improvement of 6.76\% and 2.91\% in detecting texts from different domains generated by GPT4 and Claude3, respectively.

Figures (4)

• Figure 1: The distinctions between DPIC and previous detection methods. DPIC extracts the deep intrinsic characteristics of the black-box model generated texts by decoupling the prompt and the intrinsic characteristics of the generative model.
• Figure 2: An overview of DPIC. Given a candidate text, we utilize an auxiliary LLM to reconstruct the prompt based on the candidate text. The reconstructed prompt is then used for the auxiliary LLM to obtain the regenerated text. This process aims to make the candidate and regenerated texts align with their prompts, respectively. Then, by comparing the similarity between the candidate text and the regenerated text, we can determine whether the candidate text is generated by LLMs or created by humans.
• Figure 3: Detection AUROC of DNA-GPT, DNA-GPT (supervised), DNA-GPT (prompt), and DPIC. All regeneration processes are implemented using Vicuna-7b.
• Figure 4: Cosine Similarity of the candidate text and the regenerated text obtained by different prompts.