Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Short-PHD: Detecting Short LLM-generated Text with Topological Data Analysis After Off-topic Content Insertion

Dongjun Wei, Minjia Mao, Xiao Fang, Michael Chau

TL;DR

Short-PHD tackles the challenge of detecting short LLM-generated texts by stabilizing persistence-based scores through Off-topic Content Insertion (OCI). By modeling PHD scores as normal and leveraging multiple OCI insertions to reduce estimation variance, Short-PHD achieves higher AUC than prior PHD and other zero-shot detectors on short texts, while remaining robust to common detection attacks. The approach maintains a discriminative gap between human-written and machine-generated text, enabling effective threshold-based detection in zero-shot settings. Across public and generated datasets, the results demonstrate practical viability for rapid, model-agnostic detection of short content.

Abstract

The malicious usage of large language models (LLMs) has motivated the detection of LLM-generated texts. Previous work in topological data analysis shows that the persistent homology dimension (PHD) of text embeddings can serve as a more robust and promising score than other zero-shot methods. However, effectively detecting short LLM-generated texts remains a challenge. This paper presents Short-PHD, a zero-shot LLM-generated text detection method tailored for short texts. Short-PHD stabilizes the estimation of the previous PHD method for short texts by inserting off-topic content before the given input text and identifies LLM-generated text based on an established detection threshold. Experimental results on both public and generated datasets demonstrate that Short-PHD outperforms existing zero-shot methods in short LLM-generated text detection. Implementation codes are available online.

Short-PHD: Detecting Short LLM-generated Text with Topological Data Analysis After Off-topic Content Insertion

TL;DR

Short-PHD tackles the challenge of detecting short LLM-generated texts by stabilizing persistence-based scores through Off-topic Content Insertion (OCI). By modeling PHD scores as normal and leveraging multiple OCI insertions to reduce estimation variance, Short-PHD achieves higher AUC than prior PHD and other zero-shot detectors on short texts, while remaining robust to common detection attacks. The approach maintains a discriminative gap between human-written and machine-generated text, enabling effective threshold-based detection in zero-shot settings. Across public and generated datasets, the results demonstrate practical viability for rapid, model-agnostic detection of short content.

Abstract

The malicious usage of large language models (LLMs) has motivated the detection of LLM-generated texts. Previous work in topological data analysis shows that the persistent homology dimension (PHD) of text embeddings can serve as a more robust and promising score than other zero-shot methods. However, effectively detecting short LLM-generated texts remains a challenge. This paper presents Short-PHD, a zero-shot LLM-generated text detection method tailored for short texts. Short-PHD stabilizes the estimation of the previous PHD method for short texts by inserting off-topic content before the given input text and identifies LLM-generated text based on an established detection threshold. Experimental results on both public and generated datasets demonstrate that Short-PHD outperforms existing zero-shot methods in short LLM-generated text detection. Implementation codes are available online.

Paper Structure

This paper contains 21 sections, 4 equations, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Mathematics Preliminary
  4. Estimating Persistence Homology Dimension
  5. Method: Short-PHD
  6. OCI: Off-topic Content Insertion
  7. Short-PHD: Detecting Short LLM-generated Text
  8. Empirical Validation
  9. Experiments
  10. Experimental Setup
  11. Main Results
  12. Sensitivity Analysis
  13. Conclusions and Future Work
  14. Related Work
  15. Mathematics Preliminary
  16. ...and 6 more sections

Figures (5)

  • Figure 1: PHD tulchinskii2024intrinsic v.s. Short-PHD. $n_i$ is the number of sampled points in the point cloud $W$. $W_i$ is the sampled subset of points in $W$ with size $n_i$. $E(W_{i})$ is computed by the minimal spanning tree (MST).
  • Figure 2: Boxplots of score distributions on human-written and GPT-3.5-generated Wikipedia and WritingPrompts (WP) data.
  • Figure 3: Box plots of PHD and Short-PHD distributions for texts generated by GPT-4o.
  • Figure 4: Sensitivity Analysis.
  • Figure A1: Box plots of PHD distributions by PHD and Short-PHD.