Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts

Beining Xu, Arkaitz Zubiaga

TL;DR

This work investigates how Reinforcement Learning from Human Feedback (RLHF) influences both the quality and detectability of LLM-generated text. Using a Llama-7B base model, the authors train with RLHF on two tasks—Question Answering and Instruction Following—and generate text for evaluation against two detector paradigms: training-based GPTZero and zero-shot Fast-DetectGPT. Findings show RLHF improves task-quality metrics and readability but tends to produce longer, more repetitive outputs that are easier to detect, especially for training-based detectors; zero-shot detectors prove more robust to RLHF-induced changes. The study further analyzes how output length and diversity affect detection and discusses implications for robust detection in the presence of RLHF-aligned models. The results highlight a trade-off between text quality and detectability and guide future work toward more resilient detectors against evolving RLHF-driven text generation.

Abstract

Large Language Models (LLMs) have demonstrated exceptional performance on a range of downstream NLP tasks by generating text that closely resembles human writing. However, the ease of achieving this similarity raises concerns from potential malicious uses at scale by bad actors, as LLM-generated text becomes increasingly difficult to discern from human text. Although detection methods have been developed to address this issue, bad actors can further manipulate LLM-generated texts to make them less detectable. In this work, we study how further editing texts with Reinforcement Learning from Human Feedback (RLHF), which aligns model outputs with human preferences, affects (a) the quality of generated texts for two tasks, and (b) the performance of LLM-generated text detectors, looking at both training-based and zero-shot detection methods. Although RLHF improves the quality of LLM-generated texts, we find that it also tends to produce more detectable, lengthy, and repetitive outputs. Additionally, we observe that training-based detectors are vulnerable to short texts and to texts that incorporate code, whereas zero-shot detectors exhibit greater robustness.

Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts

TL;DR

This work investigates how Reinforcement Learning from Human Feedback (RLHF) influences both the quality and detectability of LLM-generated text. Using a Llama-7B base model, the authors train with RLHF on two tasks—Question Answering and Instruction Following—and generate text for evaluation against two detector paradigms: training-based GPTZero and zero-shot Fast-DetectGPT. Findings show RLHF improves task-quality metrics and readability but tends to produce longer, more repetitive outputs that are easier to detect, especially for training-based detectors; zero-shot detectors prove more robust to RLHF-induced changes. The study further analyzes how output length and diversity affect detection and discusses implications for robust detection in the presence of RLHF-aligned models. The results highlight a trade-off between text quality and detectability and guide future work toward more resilient detectors against evolving RLHF-driven text generation.

Abstract

Large Language Models (LLMs) have demonstrated exceptional performance on a range of downstream NLP tasks by generating text that closely resembles human writing. However, the ease of achieving this similarity raises concerns from potential malicious uses at scale by bad actors, as LLM-generated text becomes increasingly difficult to discern from human text. Although detection methods have been developed to address this issue, bad actors can further manipulate LLM-generated texts to make them less detectable. In this work, we study how further editing texts with Reinforcement Learning from Human Feedback (RLHF), which aligns model outputs with human preferences, affects (a) the quality of generated texts for two tasks, and (b) the performance of LLM-generated text detectors, looking at both training-based and zero-shot detection methods. Although RLHF improves the quality of LLM-generated texts, we find that it also tends to produce more detectable, lengthy, and repetitive outputs. Additionally, we observe that training-based detectors are vulnerable to short texts and to texts that incorporate code, whereas zero-shot detectors exhibit greater robustness.

Paper Structure

This paper contains 33 sections, 2 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models (LLMs)
  4. Reinforcement Learning from Human Feedback (RLHF)
  5. LLM-Generated Text Detection
  6. Problem Formulation
  7. Model Training
  8. Pre-trained model
  9. Supervised fine-tuning
  10. Reward modeling
  11. Reinforcement learning from human feedback
  12. Tasks and Datasets
  13. Question Answering
  14. Instruction following
  15. LLM-generated Text Detector
  16. ...and 18 more sections

Figures (3)

  • Figure 1: The pipeline. This project contains three main parts: training with , evaluating the performance of the trained models, and evaluating the performance of different types of detectors.
  • Figure 2: Output sentence length of models. The x-axis represents the index of input, and the y-axis represents the average sentence length of the outputs. The number inside the bar represents the average sentence length of the output.
  • Figure 3: AUROC for detecting samples of different sentence length. The words count of (0-25) is not available for GPT-Zero since it requires the minimum input length of 150 characters. The word count of (>100) is considered in $Llama_{SFT}$ and $Llama_{PPO}$ since there are abundant outputs in these two models.