Advancing LLM detection in the ALTA 2024 Shared Task: Techniques and Analysis

TL;DR

The paper addresses reliable detection of AI-generated text in hybrid articles by evaluating sentences rather than entire documents within the ALTA 2024 Shared Task. It compares a TF-IDF Naive Bayes baseline against a fine-tuned LLaMA-3.1-8B-Instruct model, leveraging marginal token probabilities $P(\text{token}_i \mid \text{context})$ to identify distinctive patterns. Key findings show that a domain-specific, 4-bit quantized LLaMA model achieves a test Kappa of 0.932 and accuracy of 0.967, outperforming the baseline and demonstrating robustness to superficial paraphrasing. The work provides a practical approach for AI-content screening in academia and journalism while highlighting the need for broader, out-of-domain datasets to counter evolving generation strategies.

Abstract

The recent proliferation of AI-generated content has prompted significant interest in developing reliable detection methods. This study explores techniques for identifying AI-generated text through sentence-level evaluation within hybrid articles. Our findings indicate that ChatGPT-3.5 Turbo exhibits distinct, repetitive probability patterns that enable consistent in-domain detection. Empirical tests show that minor textual modifications, such as rewording, have minimal impact on detection accuracy. These results provide valuable insights for advancing AI detection methodologies, offering a pathway toward robust solutions to address the complexities of synthetic text identification.