Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stylometry Analysis of Human and Machine Text for Academic Integrity

Hezam Albaqami, Muhammad Asif Ayub, Nasir Ahmad, Yaseen Ahmad, Mohammed M. Alqahtani, Abdullah M. Algamdi, Almoaid A. Owaidah, Kashif Ahmad

TL;DR

Frames academic integrity challenges posed by AI-generated content and authorship verification, and proposes a stylometry-based NLP framework tackling four tasks, supported by Gemini-generated datasets. It evaluates machine-vs-human text classification, single- versus multi-authored documents, author-change detection, and author recognition using transformer models trained on injected machine text. The study finds near-perfect accuracy for machine-vs-human and single-vs-multi-authored tasks, while author-change detection and author recognition remain challenging, especially under strict prompts. The work provides a publicly available benchmark and analyzes prompt impact, outlining future directions in multi-modal and adversarial approaches to strengthen academic integrity.

Abstract

This work addresses critical challenges to academic integrity, including plagiarism, fabrication, and verification of authorship of educational content, by proposing a Natural Language Processing (NLP)-based framework for authenticating students' content through author attribution and style change detection. Despite some initial efforts, several aspects of the topic are yet to be explored. In contrast to existing solutions, the paper provides a comprehensive analysis of the topic by targeting four relevant tasks, including (i) classification of human and machine text, (ii) differentiating in single and multi-authored documents, (iii) author change detection within multi-authored documents, and (iv) author recognition in collaboratively produced documents. The solutions proposed for the tasks are evaluated on two datasets generated with Gemini using two different prompts, including a normal and a strict set of instructions. During experiments, some reduction in the performance of the proposed solutions is observed on the dataset generated through the strict prompt, demonstrating the complexities involved in detecting machine-generated text with cleverly crafted prompts. The generated datasets, code, and other relevant materials are made publicly available on GitHub, which are expected to provide a baseline for future research in the domain.

Stylometry Analysis of Human and Machine Text for Academic Integrity

TL;DR

Frames academic integrity challenges posed by AI-generated content and authorship verification, and proposes a stylometry-based NLP framework tackling four tasks, supported by Gemini-generated datasets. It evaluates machine-vs-human text classification, single- versus multi-authored documents, author-change detection, and author recognition using transformer models trained on injected machine text. The study finds near-perfect accuracy for machine-vs-human and single-vs-multi-authored tasks, while author-change detection and author recognition remain challenging, especially under strict prompts. The work provides a publicly available benchmark and analyzes prompt impact, outlining future directions in multi-modal and adversarial approaches to strengthen academic integrity.

Abstract

This work addresses critical challenges to academic integrity, including plagiarism, fabrication, and verification of authorship of educational content, by proposing a Natural Language Processing (NLP)-based framework for authenticating students' content through author attribution and style change detection. Despite some initial efforts, several aspects of the topic are yet to be explored. In contrast to existing solutions, the paper provides a comprehensive analysis of the topic by targeting four relevant tasks, including (i) classification of human and machine text, (ii) differentiating in single and multi-authored documents, (iii) author change detection within multi-authored documents, and (iv) author recognition in collaboratively produced documents. The solutions proposed for the tasks are evaluated on two datasets generated with Gemini using two different prompts, including a normal and a strict set of instructions. During experiments, some reduction in the performance of the proposed solutions is observed on the dataset generated through the strict prompt, demonstrating the complexities involved in detecting machine-generated text with cleverly crafted prompts. The generated datasets, code, and other relevant materials are made publicly available on GitHub, which are expected to provide a baseline for future research in the domain.
Paper Structure (17 sections, 3 figures, 9 tables)

This paper contains 17 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Dataset Creation
  5. Data Pre-processing
  6. Text Classification
  7. Experiments and Results
  8. Experimental Setup
  9. Experimental Results
  10. Machine and Human-text Classification
  11. Classification of Single and Multi-authored Documents
  12. Author Change Detection in Multi-authored Documents
  13. Author Recognition in Multi-authored Documents
  14. Conclusion and Future Work
  15. Author contributions statement
  16. ...and 2 more sections

Figures (3)

  • Figure 1: The proposed methodology.
  • Figure 2: Flowchart of the Data generation process. The same flowchart is used for both datasets, with only differences in the instruction sets.
  • Figure 3: A comparison of SBERT similarities of both datasets. Dataset 1 represents the dataset generated with the normal prompt, while Dataset 2 contains machine-generated text using the strict prompts. Moreover, class 0 and class 1 represent human and machine text, respectively.