Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI Generated Text Detection

Adilkhan Alikhanov, Aidar Amangeldi, Diar Demeubay, Dilnaz Akhmetzhan, Nurbek Moldakhmetov, Omar Polat, Galymzhan Zharas

TL;DR

The paper tackles AI-generated text detection by evaluating traditional ML, BiLSTM, and DistilBERT on a unified HC3+DAIGT v2 benchmark implemented with topic-based splits to prevent leakage. It shows that while TF-IDF logistic regression provides a solid baseline (accuracy around 82.87%), deep models, particularly DistilBERT, achieve the best overall performance (accuracy ~88.11%, ROC-AUC ~0.96), with BiLSTM offering strong contextual modeling (AUC ~0.94). The study analyzes per-class performance, training dynamics, and computational trade-offs, emphasizing the importance of contextual semantics over lexical cues. Limitations include dataset diversity and computational constraints, with future work proposing dataset expansion, adversarial robustness, and parameter-efficient tuning like LoRA to enhance generalization and practicality.

Abstract

The rapid development of large language models has led to an increase in AI-generated text, with students increasingly using LLM-generated content as their own work, which violates academic integrity. This paper presents an evaluation of AI text detection methods, including both traditional machine learning models and transformer-based architectures. We utilize two datasets, HC3 and DAIGT v2, to build a unified benchmark and apply a topic-based data split to prevent information leakage. This approach ensures robust generalization across unseen domains. Our experiments show that TF-IDF logistic regression achieves a reasonable baseline accuracy of 82.87%. However, deep learning models outperform it. The BiLSTM classifier achieves an accuracy of 88.86%, while DistilBERT achieves a similar accuracy of 88.11% with the highest ROC-AUC score of 0.96, demonstrating the strongest overall performance. The results indicate that contextual semantic modeling is significantly superior to lexical features and highlight the importance of mitigating topic memorization through appropriate evaluation protocols. The limitations of this work are primarily related to dataset diversity and computational constraints. In future work, we plan to expand dataset diversity and utilize parameter-efficient fine-tuning methods such as LoRA. We also plan to explore smaller or distilled models and employ more efficient batching strategies and hardware-aware optimization.

AI Generated Text Detection

TL;DR

The paper tackles AI-generated text detection by evaluating traditional ML, BiLSTM, and DistilBERT on a unified HC3+DAIGT v2 benchmark implemented with topic-based splits to prevent leakage. It shows that while TF-IDF logistic regression provides a solid baseline (accuracy around 82.87%), deep models, particularly DistilBERT, achieve the best overall performance (accuracy ~88.11%, ROC-AUC ~0.96), with BiLSTM offering strong contextual modeling (AUC ~0.94). The study analyzes per-class performance, training dynamics, and computational trade-offs, emphasizing the importance of contextual semantics over lexical cues. Limitations include dataset diversity and computational constraints, with future work proposing dataset expansion, adversarial robustness, and parameter-efficient tuning like LoRA to enhance generalization and practicality.

Abstract

The rapid development of large language models has led to an increase in AI-generated text, with students increasingly using LLM-generated content as their own work, which violates academic integrity. This paper presents an evaluation of AI text detection methods, including both traditional machine learning models and transformer-based architectures. We utilize two datasets, HC3 and DAIGT v2, to build a unified benchmark and apply a topic-based data split to prevent information leakage. This approach ensures robust generalization across unseen domains. Our experiments show that TF-IDF logistic regression achieves a reasonable baseline accuracy of 82.87%. However, deep learning models outperform it. The BiLSTM classifier achieves an accuracy of 88.86%, while DistilBERT achieves a similar accuracy of 88.11% with the highest ROC-AUC score of 0.96, demonstrating the strongest overall performance. The results indicate that contextual semantic modeling is significantly superior to lexical features and highlight the importance of mitigating topic memorization through appropriate evaluation protocols. The limitations of this work are primarily related to dataset diversity and computational constraints. In future work, we plan to expand dataset diversity and utilize parameter-efficient fine-tuning methods such as LoRA. We also plan to explore smaller or distilled models and employ more efficient batching strategies and hardware-aware optimization.
Paper Structure (33 sections, 2 figures, 3 tables)

This paper contains 33 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Problem Statement
  4. Related Work
  5. Traditional ML and statistical methods
  6. Deep learning based methods
  7. Datasets
  8. HC3 Dataset
  9. DAIGT v2 Dataset
  10. Topic-Based Splitting
  11. Methodology
  12. Dataset Preparation
  13. Model Architectures
  14. Logistic Regression Baseline
  15. Bidirectional LSTM
  16. ...and 18 more sections

Figures (2)

  • Figure 1: Confusion matrices for (left) Logistic Regression, (center) BiLSTM, and (right) DistilBERT on the test set.
  • Figure 2: ROC curves for BiLSTM (AUC=0.94) and DistilBERT (AUC=0.96).