On the Effectiveness of LLM-Specific Fine-Tuning for Detecting AI-Generated Text

TL;DR

This study builds a scalable benchmark for AI-generated text detection by assembling a 1B-token human corpus and a 1.9B-token AI-generated corpus across 21 LLMs, then evaluates multiple detection strategies. It introduces two fine-tuning paradigms—per-LLM and per-LLM-family—to study model-specific versus family-wide detection, achieving near-perfect token-level accuracy for strong individual detectors. An ensemble approach shows higher recall but lower precision, while per-family detectors offer a favorable balance of performance and cost. Compared to an open-source baseline, the best single detector (Phi-4 variant) and the family-based ensembles demonstrate substantial performance gains, highlighting the value of targeted fine-tuning for AI-text detection in controlled settings.

Abstract

The rapid progress of large language models has enabled the generation of text that closely resembles human writing, creating challenges for authenticity verification in education, publishing, and digital security. Detecting AI-generated text has therefore become a crucial technical and ethical issue. This paper presents a comprehensive study of AI-generated text detection based on large-scale corpora and novel training strategies. We introduce a 1-billion-token corpus of human-authored texts spanning multiple genres and a 1.9-billion-token corpus of AI-generated texts produced by prompting a variety of LLMs across diverse domains. Using these resources, we develop and evaluate numerous detection models and propose two novel training paradigms: Per LLM and Per LLM family fine-tuning. Across a 100-million-token benchmark covering 21 large language models, our best fine-tuned detector achieves up to $99.6\%$ token-level accuracy, substantially outperforming existing open-source baselines.

Figures (6)

• Figure 1: Overview of the data generation pipeline
• Figure 2: Cumulative fractions of total samples and tokens by max length
• Figure 3: AI-generated text detection pipeline for LLM fine-tuning
• Figure 4: Validation accuracy vs number of model parameters by LLM family
• Figure 5: Scatter plot of number of model parameters vs accuracy on the master-testset dataset for all conducted experiments