Argument Rarity-based Originality Assessment for AI-Assisted Writing

TL;DR

This work tackles the challenge of evaluating originality in student essays in the era of high-quality AI-generated text. It introduces AROA, a density-based, multi-component framework that quantifies argumentative originality via structural rarity, claim rarity, evidence rarity, and cognitive depth, while separately adjusting for quality. Empirical results show a strong negative relation between quality and semantic rarity, illustrating a trade-off where high-quality texts rely on common claim patterns and evidence; AI essays mimic structural complexity but lag in semantic originality. The framework demonstrates scalability, with rapid per-essay processing, modest costs, and partial transferability across LLMs, suggesting a path toward objective, large-scale originality assessment that complements traditional quality metrics. Overall, AR​OA reframes writing assessment in education from quality-centric to originality-centric, providing a practical, model-robust tool for fostering critical thinking and diverse perspectives in AI-assisted writing contexts.

Abstract

As Large Language Models (LLMs) have become capable of effortlessly generating high-quality text, traditional quality-focused writing assessment is losing its significance. If the essential goal of education is to foster critical thinking and original perspectives, assessment must also shift its paradigm from quality to originality. This study proposes Argument Rarity-based Originality Assessment (AROA), a framework for automatically evaluating argumentative originality in student essays. AROA defines originality as rarity within a reference corpus and evaluates it through four complementary components: structural rarity, claim rarity, evidence rarity, and cognitive depth. The framework quantifies the rarity of each component using density estimation and integrates them with a quality adjustment mechanism, thereby treating quality and originality as independent evaluation axes. Experiments using human essays and AI-generated essays revealed a strong negative correlation between quality and claim rarity, demonstrating a quality-originality trade-off where higher-quality texts tend to rely on typical claim patterns. Furthermore, while AI essays achieved comparable levels of structural complexity to human essays, their claim rarity was substantially lower than that of humans, indicating that LLMs can reproduce the form of argumentation but have limitations in the originality of content.

Figures (7)

• Figure 1: Overview of the AROA processing pipeline.
• Figure 2: Distribution of final originality scores ($S_{\text{final}}$) for human essays and AI essays. Dashed lines indicate the mean of each group.
• Figure 3: Comparison of four originality components between human and AI essays: Structural Rarity ($S_{\text{struct}}$), Claim Rarity ($S_{\text{claim}}$), Evidence Rarity ($S_{\text{evid}}$), and Cognitive Depth ($S_{\text{move}}$).
• Figure 4: Correlation matrix of originality components and quality score.
• Figure 5: Comparison of quality scores between human and AI essays.