Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comprehensive Dataset for Human vs. AI Generated Text Detection

Rajarshi Roy, Nasrin Imanpour, Ashhar Aziz, Shashwat Bajpai, Gurpreet Singh, Shwetangshu Biswas, Kapil Wanaskar, Parth Patwa, Subhankar Ghosh, Shreyas Dixit, Nilesh Ranjan Pal, Vipula Rawte, Ritvik Garimella, Gaytri Jena, Amit Sheth, Vasu Sharma, Aishwarya Naresh Reganti, Vinija Jain, Aman Chadha, Amitava Das

TL;DR

The paper introduces a large, annotated dataset that pairs authentic New York Times articles with AI-generated counterparts from six modern language models to advance AI-generated text detection and attribution. By using article abstracts as prompts and retrieving full human narratives via URLs, the dataset provides a realistic, diverse benchmark for distinguishing human from machine text and for identifying the likely source model. A rewriting-based Levenshtein-distance baseline demonstrates the task's difficulty, achieving 58.35% accuracy on human-vs-AI detection and 8.92% on attribution, underscoring the need for more robust methods. The resource enables cross-model generalization studies and benchmark-driven progress toward trustworthy, authentic journalism in the era of generative AI, and is publicly available for research at HuggingFace.

Abstract

The rapid advancement of large language models (LLMs) has led to increasingly human-like AI-generated text, raising concerns about content authenticity, misinformation, and trustworthiness. Addressing the challenge of reliably detecting AI-generated text and attributing it to specific models requires large-scale, diverse, and well-annotated datasets. In this work, we present a comprehensive dataset comprising over 58,000 text samples that combine authentic New York Times articles with synthetic versions generated by multiple state-of-the-art LLMs including Gemma-2-9b, Mistral-7B, Qwen-2-72B, LLaMA-8B, Yi-Large, and GPT-4-o. The dataset provides original article abstracts as prompts, full human-authored narratives. We establish baseline results for two key tasks: distinguishing human-written from AI-generated text, achieving an accuracy of 58.35\%, and attributing AI texts to their generating models with an accuracy of 8.92\%. By bridging real-world journalistic content with modern generative models, the dataset aims to catalyze the development of robust detection and attribution methods, fostering trust and transparency in the era of generative AI. Our dataset is available at: https://huggingface.co/datasets/gsingh1-py/train.

A Comprehensive Dataset for Human vs. AI Generated Text Detection

TL;DR

The paper introduces a large, annotated dataset that pairs authentic New York Times articles with AI-generated counterparts from six modern language models to advance AI-generated text detection and attribution. By using article abstracts as prompts and retrieving full human narratives via URLs, the dataset provides a realistic, diverse benchmark for distinguishing human from machine text and for identifying the likely source model. A rewriting-based Levenshtein-distance baseline demonstrates the task's difficulty, achieving 58.35% accuracy on human-vs-AI detection and 8.92% on attribution, underscoring the need for more robust methods. The resource enables cross-model generalization studies and benchmark-driven progress toward trustworthy, authentic journalism in the era of generative AI, and is publicly available for research at HuggingFace.

Abstract

The rapid advancement of large language models (LLMs) has led to increasingly human-like AI-generated text, raising concerns about content authenticity, misinformation, and trustworthiness. Addressing the challenge of reliably detecting AI-generated text and attributing it to specific models requires large-scale, diverse, and well-annotated datasets. In this work, we present a comprehensive dataset comprising over 58,000 text samples that combine authentic New York Times articles with synthetic versions generated by multiple state-of-the-art LLMs including Gemma-2-9b, Mistral-7B, Qwen-2-72B, LLaMA-8B, Yi-Large, and GPT-4-o. The dataset provides original article abstracts as prompts, full human-authored narratives. We establish baseline results for two key tasks: distinguishing human-written from AI-generated text, achieving an accuracy of 58.35\%, and attributing AI texts to their generating models with an accuracy of 8.92\%. By bridging real-world journalistic content with modern generative models, the dataset aims to catalyze the development of robust detection and attribution methods, fostering trust and transparency in the era of generative AI. Our dataset is available at: https://huggingface.co/datasets/gsingh1-py/train.
Paper Structure (15 sections, 2 figures, 2 tables)

This paper contains 15 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI-Generated Text Detection
  4. Fake News Detection
  5. Dataset Creation
  6. base data source
  7. Feature Extraction and Transformation
  8. Abstract as Prompt:
  9. Web URL for Human Story Extraction:
  10. Generation of AI Texts
  11. Dataset Structure and Statistics
  12. Potential uses of the dataset
  13. Baseline
  14. Results
  15. Conclusion and Future Work

Figures (2)

  • Figure 1: Wordcloud of human written and AI generated text.
  • Figure 2: Illustration of Raidar concept. Given a Yelp review, GPT-3.5-Turbo is asked to rewrite the input text while preserving meaning. The rewriting of a human-written review undergoes more character-level edits (highlighted in red/orange), while the rewriting of a GPT-written review remains largely unchanged.