Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluation Framework for AI Creativity: A Case Study Based on Story Generation

Pharath Sathya, Yin Jou Huang, Fei Cheng

TL;DR

Addressing the challenge of evaluating AI-generated creative text, the paper proposes a domain-specific, four-component framework (Novelty, Value, Adherence, Resonance) and validates it with Spike Prompting and crowdsourced data ($N=115$). It reveals that creativity judgments are hierarchical and that reflective processing reshapes ratings and inter-rater agreement, exposing dimensions obscured by reference-based metrics. It also shows a Creativity–Enjoyment dissociation and a polarization phenomenon in mid-range scores, arguing for multi-dimensional, audience-aware evaluation practices. The framework has practical implications for developing and benchmarking AI storytelling systems beyond traditional gold-standard proximity.

Abstract

Evaluating creative text generation remains a challenge because existing reference-based metrics fail to capture the subjective nature of creativity. We propose a structured evaluation framework for AI story generation comprising four components (Novelty, Value, Adherence, and Resonance) and eleven sub-components. Using controlled story generation via ``Spike Prompting'' and a crowdsourced study of 115 readers, we examine how different creative components shape both immediate and reflective human creativity judgments. Our findings show that creativity is evaluated hierarchically rather than cumulatively, with different dimensions becoming salient at different stages of judgment, and that reflective evaluation substantially alters both ratings and inter-rater agreement. Together, these results support the effectiveness of our framework in revealing dimensions of creativity that are obscured by reference-based evaluation.

Evaluation Framework for AI Creativity: A Case Study Based on Story Generation

TL;DR

Addressing the challenge of evaluating AI-generated creative text, the paper proposes a domain-specific, four-component framework (Novelty, Value, Adherence, Resonance) and validates it with Spike Prompting and crowdsourced data (). It reveals that creativity judgments are hierarchical and that reflective processing reshapes ratings and inter-rater agreement, exposing dimensions obscured by reference-based metrics. It also shows a Creativity–Enjoyment dissociation and a polarization phenomenon in mid-range scores, arguing for multi-dimensional, audience-aware evaluation practices. The framework has practical implications for developing and benchmarking AI storytelling systems beyond traditional gold-standard proximity.

Abstract

Evaluating creative text generation remains a challenge because existing reference-based metrics fail to capture the subjective nature of creativity. We propose a structured evaluation framework for AI story generation comprising four components (Novelty, Value, Adherence, and Resonance) and eleven sub-components. Using controlled story generation via ``Spike Prompting'' and a crowdsourced study of 115 readers, we examine how different creative components shape both immediate and reflective human creativity judgments. Our findings show that creativity is evaluated hierarchically rather than cumulatively, with different dimensions becoming salient at different stages of judgment, and that reflective evaluation substantially alters both ratings and inter-rater agreement. Together, these results support the effectiveness of our framework in revealing dimensions of creativity that are obscured by reference-based evaluation.
Paper Structure (64 sections, 5 figures, 8 tables)

This paper contains 64 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Creativity Components
  4. Story Generation
  5. Readability Control
  6. Human Evaluation Procedure
  7. Results
  8. Measurement Validation
  9. Internal Consistency.
  10. Construct Validity.
  11. Which Creativity Dimensions Matter First? A Hierarchical Analysis
  12. A Reversal in Evaluation Order.
  13. The Re-emergence of Adherence.
  14. The Instability of Agreement: Reflection Impact & Overall User Agreement
  15. The Reflection Gap.
  16. ...and 49 more sections

Figures (5)

  • Figure 1: Reversal in Evaluation Order. Feature importance comparison between Initial (Blue) and Reflective (Red) creativity.
  • Figure 2: The Subjectivity Trap.Left: The Reflection Gap shows that analytical thinking boosts Technical stories (Clinical Tone: High Adherence) but hurts Creative ones (Melancholic/Witty). Right: The Consensus U-Curve shows that mid-range scores mask high polarization (high variance).
  • Figure 3: The Shifting Formula. Sub-component weights across outcome variables. Note the "Empathy Gap": Empathy (purple bar) drives Enjoyment but vanishes for Creativity. Conversely, Vocabulary (blue/red bars) drives Creativity but is irrelevant for Enjoyment.
  • Figure 4: Informed Consent Form. Screenshot of the consent page shown to participants prior to accessing any study materials. Participants were required to explicitly agree before proceeding.
  • Figure 5: Task Comprehension Check. Screenshot of the instruction page shown after consent. Participants were required to confirm task understanding before proceeding; selecting "No" terminated the survey.