Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts

TL;DR

This paper tackles the problem of robustly detecting AI-generated text across unseen models and domains by estimating the intrinsic dimensionality of text embeddings with the persistent homology dimension (PHD). The authors show that human texts exhibit higher intrinsic dimension (approximately $d_{human} \approx 9$ for alphabetic languages and $d_{human} \approx 7$ for Chinese) while AI-generated texts are about $1.5$ lower on average, enabling a simple, model-agnostic detector. They develop an efficient PHD-based estimator, apply it to contextual embeddings from RoBERTa/XLM-R and use the resulting score in a single-feature logistic classifier, achieving strong cross-domain and cross-model performance, including resilience to paraphrase attacks. A multilingual dataset release and bias-reduction results against non-native writers further underscore the practical impact and versatility of the approach, suggesting a promising direction for topological data analysis in natural language processing.

Abstract

Rapidly increasing quality of AI-generated content makes it difficult to distinguish between human and AI-generated texts, which may lead to undesirable consequences for society. Therefore, it becomes increasingly important to study the properties of human texts that are invariant over different text domains and varying proficiency of human writers, can be easily calculated for any language, and can robustly separate natural and AI-generated texts regardless of the generation model and sampling method. In this work, we propose such an invariant for human-written texts, namely the intrinsic dimensionality of the manifold underlying the set of embeddings for a given text sample. We show that the average intrinsic dimensionality of fluent texts in a natural language is hovering around the value $9$ for several alphabet-based languages and around $7$ for Chinese, while the average intrinsic dimensionality of AI-generated texts for each language is $\approx 1.5$ lower, with a clear statistical separation between human-generated and AI-generated distributions. This property allows us to build a score-based artificial text detector. The proposed detector's accuracy is stable over text domains, generator models, and human writer proficiency levels, outperforming SOTA detectors in model-agnostic and cross-domain scenarios by a significant margin.

Figures (10)

• Figure 1: Real and artificial text have different intrinsic dimension: (a-b) idea; (c) actual results.
• Figure 2: A comparison of ID estimators with noise on artificial datasets; lower is better.
• Figure 3: Boxplots of PHD distributions for different generative models in comparison to human-written text on Wikipedia data. Embeddings are obtained from RoBERTa-base.
• Figure 4: Boxplots of PHD distributions in different languages on Wikipedia data. Embeddings are obtained from XLM-RoBERTa-base (multilingual).
• Figure 6: Comparison of GPT detectors in non-standard environment. Left: bias against non-native English writing samples (the lower is better). Right: effect of the prompt design on performance (the higher is better).