Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Subtle Differences between Human and Model Languages Using Spectrum of Relative Likelihood

Yang Xu, Yu Wang, Hao An, Zhichen Liu, Yongyuan Li

TL;DR

This work tackles the growing challenge of distinguishing human-written from model-generated text as language models become increasingly human-like. It introduces FourierGPT, which converts tokenwise relative likelihoods into a spectrum via a Fourier transform and uses both supervised and heuristic classifiers to detect authorship. The approach achieves competitive zero-shot performance and state-of-the-art results on short-text detection, while requiring less computational cost by using relative likelihood and simple estimators such as $n$-gram models. Its findings also reveal subtle linguistic differences—such as distinctive Yes/No answer patterns and robustness to POS masking—grounded in psycholinguistic concepts like surprisal and UID, with practical implications for rapid, scalable text attribution. The work provides open-source code and demonstrates that spectrum-based likelihood analysis can offer robust, interpretable signals even when absolute likelihood estimates are noisy or model-dependent.

Abstract

Human and model-generated texts can be distinguished by examining the magnitude of likelihood in language. However, it is becoming increasingly difficult as language model's capabilities of generating human-like texts keep evolving. This study provides a new perspective by using the relative likelihood values instead of absolute ones, and extracting useful features from the spectrum-view of likelihood for the human-model text detection task. We propose a detection procedure with two classification methods, supervised and heuristic-based, respectively, which results in competitive performances with previous zero-shot detection methods and a new state-of-the-art on short-text detection. Our method can also reveal subtle differences between human and model languages, which find theoretical roots in psycholinguistics studies. Our code is available at https://github.com/CLCS-SUSTech/FourierGPT

Detecting Subtle Differences between Human and Model Languages Using Spectrum of Relative Likelihood

TL;DR

This work tackles the growing challenge of distinguishing human-written from model-generated text as language models become increasingly human-like. It introduces FourierGPT, which converts tokenwise relative likelihoods into a spectrum via a Fourier transform and uses both supervised and heuristic classifiers to detect authorship. The approach achieves competitive zero-shot performance and state-of-the-art results on short-text detection, while requiring less computational cost by using relative likelihood and simple estimators such as -gram models. Its findings also reveal subtle linguistic differences—such as distinctive Yes/No answer patterns and robustness to POS masking—grounded in psycholinguistic concepts like surprisal and UID, with practical implications for rapid, scalable text attribution. The work provides open-source code and demonstrates that spectrum-based likelihood analysis can offer robust, interpretable signals even when absolute likelihood estimates are noisy or model-dependent.

Abstract

Human and model-generated texts can be distinguished by examining the magnitude of likelihood in language. However, it is becoming increasingly difficult as language model's capabilities of generating human-like texts keep evolving. This study provides a new perspective by using the relative likelihood values instead of absolute ones, and extracting useful features from the spectrum-view of likelihood for the human-model text detection task. We propose a detection procedure with two classification methods, supervised and heuristic-based, respectively, which results in competitive performances with previous zero-shot detection methods and a new state-of-the-art on short-text detection. Our method can also reveal subtle differences between human and model languages, which find theoretical roots in psycholinguistics studies. Our code is available at https://github.com/CLCS-SUSTech/FourierGPT
Paper Structure (23 sections, 3 equations, 12 figures, 8 tables)

This paper contains 23 sections, 3 equations, 12 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Likelihood-based zero-shot text detection
  4. Surprisal and likelihood of language
  5. Evaluation of natural language generation with likelihood
  6. Method
  7. Estimation and normalization of likelihood scores
  8. Fourier transform
  9. Classification methods
  10. Supervised learning-based classifier
  11. Pair-wise Heuristic-based classifier
  12. Experiment Results
  13. Datasets
  14. Supervised learning-based classification
  15. Pair-wise heuristic-based classification
  16. ...and 8 more sections

Figures (12)

  • Figure 1: The procedure (above) and example (below) of FourierGPT.
  • Figure 2: Heuristics for constructing pair-wise classifiers: Likelihood spectrum shows salient difference at low frequency components. Curves are fit using generative additive models (GAM). Shaded areas are 95% confidence intervals from bootstrap.
  • Figure 3: (a) The changes of likelihood spectrum ((a) left) and likelihood-position plot ((a) right) after removing the "Yes"/"No" in answer from PubMed data (estimated by GPT-4 only). (b) The changes of spectrum estimated by all three models. Curves are fit with GAM. Shaded areas are 95% confidence intervals from bootstrap.
  • Figure 4: Text lengths affect likelihood spectrum and pairwise classifier performance. Each plot corresponds to lengths of text, $n=50,100,150$, compared to "Full". Curves are fit with GAM. Shaded areas are 95% confidence intervals from bootstrap.
  • Figure 5: Likelihood spectrum before and after masking out the 'NOUN+VERB+ADJ' POS tags. It can be seen that GPT-4 texts show greater variation before and after the mask, while human text show smaller change. Likelihood is estimated with a bigram model. Curves are fit with GAM. Shaded areas are 95% confidence intervals from bootstrap.
  • ...and 7 more figures