Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring AI "Slop" in Text

Chantal Shaib, Tuhin Chakrabarty, Diego Garcia-Olano, Byron C. Wallace

TL;DR

This work addresses the lack of a formal, measurable definition of AI-generated 'slop' by introducing a taxonomy organized into Information Utility, Information Quality, and Style Quality, and validating it with fine-grained, span-level expert annotations on 150 news articles and 100 retrieval QA passages. The study shows that binary 'slop' judgments are somewhat subjective but correlate with latent dimensions such as relevance, coherence, and factuality, with domain-specific differences in which axes matter most. Automatic measurement approaches partially capture slop signals (AUPRC ≈ 0.52–0.55; WQRM correlations), but LLMs as judges or span extractors struggle to fully replicate human assessments, highlighting limitations of current automatic metrics and reward models. The taxonomy and annotated data provide a framework for domain-aware evaluation of AI-generated text and guide future improvements in automatic slop detection and text-quality modeling.

Abstract

AI "slop" is an increasingly popular term used to describe low-quality AI-generated text, but there is currently no agreed upon definition of this term nor a means to measure its occurrence. In this work, we develop a taxonomy of "slop" through interviews with experts in NLP, writing, and philosophy, and propose a set of interpretable dimensions for its assessment in text. Through span-level annotation, we find that binary "slop" judgments are (somewhat) subjective, but such determinations nonetheless correlate with latent dimensions such as coherence and relevance. Our framework can be used to evaluate AI-generated text in both detection and binary preference tasks, potentially offering new insights into the linguistic and stylistic factors that contribute to quality judgments.

Measuring AI "Slop" in Text

TL;DR

This work addresses the lack of a formal, measurable definition of AI-generated 'slop' by introducing a taxonomy organized into Information Utility, Information Quality, and Style Quality, and validating it with fine-grained, span-level expert annotations on 150 news articles and 100 retrieval QA passages. The study shows that binary 'slop' judgments are somewhat subjective but correlate with latent dimensions such as relevance, coherence, and factuality, with domain-specific differences in which axes matter most. Automatic measurement approaches partially capture slop signals (AUPRC ≈ 0.52–0.55; WQRM correlations), but LLMs as judges or span extractors struggle to fully replicate human assessments, highlighting limitations of current automatic metrics and reward models. The taxonomy and annotated data provide a framework for domain-aware evaluation of AI-generated text and guide future improvements in automatic slop detection and text-quality modeling.

Abstract

AI "slop" is an increasingly popular term used to describe low-quality AI-generated text, but there is currently no agreed upon definition of this term nor a means to measure its occurrence. In this work, we develop a taxonomy of "slop" through interviews with experts in NLP, writing, and philosophy, and propose a set of interpretable dimensions for its assessment in text. Through span-level annotation, we find that binary "slop" judgments are (somewhat) subjective, but such determinations nonetheless correlate with latent dimensions such as coherence and relevance. Our framework can be used to evaluate AI-generated text in both detection and binary preference tasks, potentially offering new insights into the linguistic and stylistic factors that contribute to quality judgments.

Paper Structure

This paper contains 42 sections, 17 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Defining LLM "Slop"
  4. Datasets
  5. Pilot "Slop" Annotation
  6. Finalized "Slop" Taxonomy
  7. Annotating for "Slop" in Text
  8. What is "Sloppy" Text?
  9. Results
  10. Automatically Measuring Slop
  11. Linear Models
  12. Text Quality Reward Models
  13. Can LLMs Self Identify "Slop"?
  14. LLM-Generated Span Rationales
  15. Training Span Extraction Models
  16. ...and 27 more sections

Figures (17)

  • Figure 1: Sample of annotations over a human-written news article highlighting indicators of "slop" (red; from Russell2025PeopleWF), human-writing (green; ours), and both (yellow). "Slop" labels are notably different than indicators of human-written text.
  • Figure 2: All data
  • Figure 3: News
  • Figure 4: MS MARCO
  • Figure 6: Frequency of (a) human-assigned slop code prevalence and, (b) "slop" category collapsed trigrams in o3-mini span rationales in the News domain. o3-mini over-represents issues related to Density in the rationales, while under-representing issues with Coherence, Tone, Bias, Relevance, and Factuality.
  • ...and 12 more figures