Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better

TL;DR

This work tackles the challenge of detecting machine-generated text by overcoming the generalization gaps of zero-shot detectors like DetectGPT. It introduces Pecola, a two-stage detector that combines a token-level selective perturbation scheme with token-weighted, multi-pairwise contrastive learning to bridge metric-based and fine-tuned detection. Empirical results across four public datasets in few-shot settings show Pecola consistently outperforms both baselines and SOTA methods, with notable gains in robustness and cross-domain/generalization scenarios. The approach reduces reliance on thresholding, generalizes across mask-filling models, and improves short-text detection, highlighting its practical impact for robust MGT defense in diverse deployment contexts.

Abstract

The burgeoning generative capabilities of large language models (LLMs) have raised growing concerns about abuse, demanding automatic machine-generated text detectors. DetectGPT, a zero-shot metric-based detector, first introduces perturbation and shows great performance improvement. However, in DetectGPT, the random perturbation strategy could introduce noise, and logit regression depends on the threshold, harming the generalizability and applicability of individual or small-batch inputs. Hence, we propose a novel fine-tuned detector, Pecola, bridging metric-based and fine-tuned methods by contrastive learning on selective perturbation. Selective strategy retains important tokens during perturbation and weights for multi-pair contrastive learning. The experiments show that Pecola outperforms the state-of-the-art (SOTA) by 1.20% in accuracy on average on four public datasets. And we further analyze the effectiveness, robustness, and generalization of the method.

Figures (6)

• Figure 1: Example of the selective strategy perturbation of Pecola, which prevent modifying important tokens (in green). Orange tokens are the perturbed texts.
• Figure 2: Overview of Pecola. In the Selective Strategy Perturbation stage (Sec. \ref{['sec:Per']}), we use the YAKE algorithm to score token importance and then selective masking based on probability. Then, we fill in the masks with a mark-filling language model. In the Contrastive Learning stage (Sec. \ref{['sec:mcl']}), we design a multi-pairwise method with token-level weights also from tokens importance. Yellow arrows represent attraction and blue ones represent repulsion. The model is optimized by combining cross-entropy (CE) loss $\mathcal{L}_{\text{ce}}$ and contrastive loss $\mathcal{L}_{\text{con}}$. * Our method, different from DetectGPT, is generalizable on any mask-filling language model.
• Figure 3: Result of generalizing on various mask-filling models.
• Figure 4: Performance of Pecola and RoBERTa to detect shorter texts. The average token number of the original GPT-2 and HC3 datasets are 445 and 260.
• Figure 5: Impact of varying the number of perturbations and mask gap in Pecola, we use T5-large raffel2020exploring as the mask-filling model. For each combination, we conduct tests on ten randomly select seeds.