Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding the Effects of Human-written Paraphrases in LLM-generated Text Detection

Hiu Ting Lau, Arkaitz Zubiaga

TL;DR

The results show that the inclusion of human-written paraphrases has a significant impact of LLM-generated detector performance, promoting TPR@1%FPR with a possible trade-off of AUROC and accuracy.

Abstract

Natural Language Generation has been rapidly developing with the advent of large language models (LLMs). While their usage has sparked significant attention from the general public, it is important for readers to be aware when a piece of text is LLM-generated. This has brought about the need for building models that enable automated LLM-generated text detection, with the aim of mitigating potential negative outcomes of such content. Existing LLM-generated detectors show competitive performances in telling apart LLM-generated and human-written text, but this performance is likely to deteriorate when paraphrased texts are considered. In this study, we devise a new data collection strategy to collect Human & LLM Paraphrase Collection (HLPC), a first-of-its-kind dataset that incorporates human-written texts and paraphrases, as well as LLM-generated texts and paraphrases. With the aim of understanding the effects of human-written paraphrases on the performance of state-of-the-art LLM-generated text detectors OpenAI RoBERTa and watermark detectors, we perform classification experiments that incorporate human-written paraphrases, watermarked and non-watermarked LLM-generated documents from GPT and OPT, and LLM-generated paraphrases from DIPPER and BART. The results show that the inclusion of human-written paraphrases has a significant impact of LLM-generated detector performance, promoting TPR@1%FPR with a possible trade-off of AUROC and accuracy.

Understanding the Effects of Human-written Paraphrases in LLM-generated Text Detection

TL;DR

The results show that the inclusion of human-written paraphrases has a significant impact of LLM-generated detector performance, promoting TPR@1%FPR with a possible trade-off of AUROC and accuracy.

Abstract

Natural Language Generation has been rapidly developing with the advent of large language models (LLMs). While their usage has sparked significant attention from the general public, it is important for readers to be aware when a piece of text is LLM-generated. This has brought about the need for building models that enable automated LLM-generated text detection, with the aim of mitigating potential negative outcomes of such content. Existing LLM-generated detectors show competitive performances in telling apart LLM-generated and human-written text, but this performance is likely to deteriorate when paraphrased texts are considered. In this study, we devise a new data collection strategy to collect Human & LLM Paraphrase Collection (HLPC), a first-of-its-kind dataset that incorporates human-written texts and paraphrases, as well as LLM-generated texts and paraphrases. With the aim of understanding the effects of human-written paraphrases on the performance of state-of-the-art LLM-generated text detectors OpenAI RoBERTa and watermark detectors, we perform classification experiments that incorporate human-written paraphrases, watermarked and non-watermarked LLM-generated documents from GPT and OPT, and LLM-generated paraphrases from DIPPER and BART. The results show that the inclusion of human-written paraphrases has a significant impact of LLM-generated detector performance, promoting TPR@1%FPR with a possible trade-off of AUROC and accuracy.

Paper Structure

This paper contains 36 sections, 2 equations, 34 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Natural Language Generation (NLG)
  4. LLMs in NLG.
  5. Paraphrase Generation.
  6. Importance of LLM-generated Text Detection
  7. Intentional malicious usage.
  8. Ethical concerns.
  9. Information inaccuracy.
  10. State-of-the-art LLM-generated Text Detection
  11. Approaches to LLM-generated text detection.
  12. Paraphrase attacks in LLM-generated text detection.
  13. Research Gap
  14. Dataset
  15. Creating the HLPC Dataset
  16. ...and 21 more sections

Figures (34)

  • Figure 1: Flowchart of the LLM text generation, paraphrasing and classification process.
  • Figure 2: Mean similarity scores of watermarked (left) and non-watermarked (right) LLM-generated texts and their paraphrases across different datasets.
  • Figure 3: ROC Curve Comparison: (a) Human-generated Documents vs Non-watermarked LLM-generated Documents, (b) Human-generated Paraphrases vs Non-watermarked LLM-generated Documents, (c) Human-generated Documents vs Watermarked LLM-generated Documents, (d) Human-generated Paraphrases vs Watermark LLM-generated Documents
  • Figure 4: ROC Curve Comparison: (a) Human-generated Documents/Paraphrases vs DIPPER-generated Paraphrases from Non-watermarked MRPC GPT2-generated Text, (b) Human-generated Documents/Paraphrases vs BART-generated Paraphrases from Non-watermarked MRPC GPT2-generated Text, (c) Human-generated Documents/Paraphrases vs DIPPER-generated Paraphrases from Watermarked MRPC GPT2-generated Text, (d) Human-generated Documents/Paraphrases vs BART-generated Paraphrases from Watermarked MRPC GPT2-generated Text
  • Figure 5: ROC Curve from OpenAI Detector with Human-generated Documents / Paraphrases and Non-watermarked DIPPER (left) / BART-generated (right) Paraphrases from MRPC OPT-Generated Text
  • ...and 29 more figures