Attribution Quality in AI-Generated Content:Benchmarking Style Embeddings and LLM Judges

TL;DR

This work benchmarks two attribution mechanisms—fixed Style Embeddings and an instruction-tuned LLM judge—on the Human–AI Parallel Corpus to assess authorship provenance of AI-generated text. The findings reveal complementary domain strengths: Style Embeddings excel in spoken and scripted domains, while the LLM judge better handles fiction and academic prose, though results vary with the generator model ($0.82\rightarrow0.68$ vs $0.85\rightarrow0.81$ in aggregate across GPT vs Llama). McNemar’s test confirms some domain- and generator-specific significances, and the study advocates hybrid attribution frameworks that integrate structural style cues with content-aware semantic judgments. The authors release the dataset and code under the MIT license, enabling reproducibility and extension to broader LLM families and domains. Overall, attribution in AI-generated content benefits from combining complementary signals to robustly infer provenance across genres and model families.

Abstract

Attributing authorship in the era of large language models (LLMs) is increasingly challenging as machine-generated prose rivals human writing. We benchmark two complementary attribution mechanisms , fixed Style Embeddings and an instruction-tuned LLM judge (GPT-4o) on the Human AI Parallel Corpus, an open dataset of 600 balanced instances spanning six domains (academic, news, fiction, blogs, spoken transcripts, and TV/movie scripts). Each instance contains a human prompt with both a gold continuation and an LLM-generated continuation from either GPT-4o or LLaMA-70B-Instruct. The Style Embedding baseline achieves stronger aggregate accuracy on GPT continuations (82 pct vs. 68 pct). The LLM Judge is slightly better than the Style embeddings on LLaMA continuations (85 pct vs. 81 pct) but the results are not statistically significant. Crucially, the LLM judge significantly outperforms in fiction and academic prose, indicating semantic sensitivity, whereas embeddings dominate in spoken and scripted dialogue, reflecting structural strengths. These complementary patterns highlight attribution as a multidimensional problem requiring hybrid strategies. To support reproducibility we provide code on GitHub and derived data on Hugging Face under the MIT license. This open framework provides a reproducible benchmark for attribution quality assessment in AI-generated content, along with a review of related literature influencing this work.

Figures (3)

• Figure 1: Task Formulation
• Figure 2: Style Embeddings. Binary human vs. LLM continuation detection.
• Figure 3: LLM as a Judge. Binary human vs. LLM continuation detection.