Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpretable Predictability-Based AI Text Detection: A Replication Study

Adam Skurla, Dominik Macko, Jakub Simko

Abstract

This paper replicates and extends the system used in the AuTexTification 2023 shared task for authorship attribution of machine-generated texts. First, we tried to reproduce the original results. Exact replication was not possible because of differences in data splits, model availability, and implementation details. Next, we tested newer multilingual language models and added 26 document-level stylometric features. We also applied SHAP analysis to examine which features influence the model's decisions. We replaced the original GPT-2 models with newer generative models such as Qwen and mGPT for computing probabilistic features. For contextual representations, we used mDeBERTa-v3-base and applied the same configuration to both English and Spanish. This allowed us to use one shared configuration for Subtask 1 and Subtask 2. Our experiments show that the additional stylometric features improve performance in both tasks and both languages. The multilingual configuration achieves the results that are comparable to or better than language-specific models. The study also shows that clear documentation is important for reliable replication and fair comparison of systems.

Interpretable Predictability-Based AI Text Detection: A Replication Study

Abstract

This paper replicates and extends the system used in the AuTexTification 2023 shared task for authorship attribution of machine-generated texts. First, we tried to reproduce the original results. Exact replication was not possible because of differences in data splits, model availability, and implementation details. Next, we tested newer multilingual language models and added 26 document-level stylometric features. We also applied SHAP analysis to examine which features influence the model's decisions. We replaced the original GPT-2 models with newer generative models such as Qwen and mGPT for computing probabilistic features. For contextual representations, we used mDeBERTa-v3-base and applied the same configuration to both English and Spanish. This allowed us to use one shared configuration for Subtask 1 and Subtask 2. Our experiments show that the additional stylometric features improve performance in both tasks and both languages. The multilingual configuration achieves the results that are comparable to or better than language-specific models. The study also shows that clear documentation is important for reliable replication and fair comparison of systems.
Paper Structure (16 sections, 5 equations, 5 figures, 13 tables)

This paper contains 16 sections, 5 equations, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. RQ1: Reproduction of the Original System
  5. RQ2: Impact of Base Language Models
  6. RQ3: Effect of Additional Stylometric Features
  7. Discussion and Conclusion
  8. Original Study To Be Replicated
  9. System Overview
  10. Methodology
  11. Additional Stylometric Features
  12. Additional results
  13. New Configurations
  14. Robustness and Time Complexity
  15. Comparison with State-of-the-Art Detector
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Overview of the architecture proposed by przybyla2023ve. The upper part shows the evaluated model configurations, while the lower part illustrates the feature extraction pipeline used to derive token-level probabilistic features and document-level linguistic features.
  • Figure 2: SHAP Summary for Subtask 1 (English) - LingRF Style, Class 1
  • Figure 3: SHAP Summary for Subtask 2 (Spanish) - LingRF Style, Class 5
  • Figure 4: SHAP Summary for Subtask 1 (English) - LingRF Style + PredOut, Class 1
  • Figure 5: SHAP Summary for Subtask 2 (Spanish) - LingRF Style + PredOut, Class 5