Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Intuition to Expertise: Rubric-Based Cognitive Calibration for Human Detection of LLM-Generated Korean Text

Shinwoo Park, Yo-Sub Han

TL;DR

This paper tackles the challenge of differentiating human-written from LLM-generated Korean text, where fluent surface form can mislead even linguistically trained readers. It introduces LREAD, a rubric grounded in national Korean writing standards, and implements a three-phase longitudinal protocol that moves expert judgment from intuition to rubric-guided analysis and domain mastery. Results show detection accuracy improving from 60% to 100% across phases, with Fleiss' $f

Abstract

Distinguishing human-written Korean text from fluent LLM outputs remains difficult even for linguistically trained readers, who can over-trust surface well-formedness. We study whether expert detection can be treated as a learnable skill and improved through structured calibration. We introduce LREAD, a rubric derived from national Korean writing standards and adapted to target micro-level artifacts (e.g., punctuation optionality, spacing behavior, and register shifts). In a three-phase longitudinal blind protocol with Korean linguistics majors, Phase 1 measures intuition-only detection, Phase 2 enforces criterion-level scoring with explicit justifications, and Phase 3 evaluates domain-focused mastery on held-out elementary essays. Across phases, majority-vote accuracy increases from 60% to 100%, accompanied by stronger inter-annotator agreement (Fleiss' kappa: -0.09 --> 0.82). Compared to state-of-the-art LLM detectors, calibrated humans rely more on language-specific micro-diagnostics that are not well captured by coarse discourse priors. Our findings suggest that rubric-scaffolded expert judgment can serve as an interpretable complement to automated detectors for non-English settings, and we release the full rubric and a taxonomy of calibrated detection signatures.

From Intuition to Expertise: Rubric-Based Cognitive Calibration for Human Detection of LLM-Generated Korean Text

TL;DR

This paper tackles the challenge of differentiating human-written from LLM-generated Korean text, where fluent surface form can mislead even linguistically trained readers. It introduces LREAD, a rubric grounded in national Korean writing standards, and implements a three-phase longitudinal protocol that moves expert judgment from intuition to rubric-guided analysis and domain mastery. Results show detection accuracy improving from 60% to 100% across phases, with Fleiss' $f

Abstract

Distinguishing human-written Korean text from fluent LLM outputs remains difficult even for linguistically trained readers, who can over-trust surface well-formedness. We study whether expert detection can be treated as a learnable skill and improved through structured calibration. We introduce LREAD, a rubric derived from national Korean writing standards and adapted to target micro-level artifacts (e.g., punctuation optionality, spacing behavior, and register shifts). In a three-phase longitudinal blind protocol with Korean linguistics majors, Phase 1 measures intuition-only detection, Phase 2 enforces criterion-level scoring with explicit justifications, and Phase 3 evaluates domain-focused mastery on held-out elementary essays. Across phases, majority-vote accuracy increases from 60% to 100%, accompanied by stronger inter-annotator agreement (Fleiss' kappa: -0.09 --> 0.82). Compared to state-of-the-art LLM detectors, calibrated humans rely more on language-specific micro-diagnostics that are not well captured by coarse discourse priors. Our findings suggest that rubric-scaffolded expert judgment can serve as an interpretable complement to automated detectors for non-English settings, and we release the full rubric and a taxonomy of calibrated detection signatures.
Paper Structure (40 sections, 6 figures, 9 tables)

This paper contains 40 sections, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Intuition and the Detection Gap
  3. Experimental Setup
  4. Human-LLM Detection Gap
  5. Analysis: Initial Human Perception
  6. LREAD: A Human-Centered Cognitive Calibration Framework
  7. Rubric Design from Linguistic Standards
  8. Three-Phase Cognitive Calibration
  9. Experimental Procedure
  10. Results of Human Calibration
  11. Accuracy Gains Across Calibration Phases
  12. Error Dynamics and Calibration
  13. Inter-annotator Agreement Analysis
  14. Machine Perception vs. Human Insight
  15. Human–LLM Comparison Protocol
  16. ...and 25 more sections

Figures (6)

  • Figure 1: Detection accuracy across the three phases for each human annotator and their majority vote.
  • Figure 2: Confusion matrices for the human-annotator majority vote in Phase 1 (intuition-only) and Phase 2 (rubric-guided). Each cell reports the number of essays in the corresponding true-label and predicted-label pair under the binary Human versus AI decision.
  • Figure 3: Phase 2 rubric profile by generator. The radar chart reports the mean macro scores (Content, Organization, Expression) aggregated from the rubric dimensions and normalized by each category maximum (40/10/50).
  • Figure 4: The full LREAD rubric with anchored scoring criteria (Korean original).
  • Figure 5: The full LREAD rubric with anchored scoring criteria (English translation).
  • ...and 1 more figures