AI Content Self-Detection for Transformer-based Large Language Models

TL;DR

This paper investigates whether transformer-based large language models can self-detect their own output to address AI-authorship attribution. It proposes self-detection as an origin-detection approach and tests three leading models (ChatGPT, Bard, Claude) via zero-shot prompts on original, paraphrased, and human-written texts. The results show model-dependent success: Bard exhibits the strongest self-detection, ChatGPT performs reasonably on its own outputs but struggles with paraphrased text, and Claude performs poorly on original output yet can detect some paraphrased content. The study highlights the potential and limitations of self-detection, underlining the need for larger-scale evaluations and robust detectors to support academic integrity in the era of AI-generated text.

Abstract

$ $The usage of generative artificial intelligence (AI) tools based on large language models, including ChatGPT, Bard, and Claude, for text generation has many exciting applications with the potential for phenomenal productivity gains. One issue is authorship attribution when using AI tools. This is especially important in an academic setting where the inappropriate use of generative AI tools may hinder student learning or stifle research by creating a large amount of automatically generated derivative work. Existing plagiarism detection systems can trace the source of submitted text but are not yet equipped with methods to accurately detect AI-generated text. This paper introduces the idea of direct origin detection and evaluates whether generative AI systems can recognize their output and distinguish it from human-written texts. We argue why current transformer-based models may be able to self-detect their own generated text and perform a small empirical study using zero-shot learning to investigate if that is the case. Results reveal varying capabilities of AI systems to identify their generated text. Google's Bard model exhibits the largest capability of self-detection with an accuracy of 94\%, followed by OpenAI's ChatGPT with 83\%. On the other hand, Anthropic's Claude model seems to be not able to self-detect.

Figures (2)

• Figure 1: AI self-detection accuracy with 95% confidence interval.
• Figure 2: AI self-detection vs. detection by other models accuracy with 95% confidence interval.