LLM Review: Enhancing Creative Writing via Blind Peer Review Feedback

TL;DR

This work addresses the challenge of sustaining creativity in large language models when used in multi-agent settings, where increased interaction can lead to homogenization. It proposes Blind Peer Review as a structured information-flow constraint and SciFi-100 as a unified evaluation framework combining LLM-as-a-judge scoring, human judgments, and rule-based novelty metrics. Empirical results show that LLM Review outperforms baselines, with smaller models sometimes surpassing larger single-agent models, suggesting interaction structure can substitute for scale for creative tasks. The findings have practical implications for designing more diverse and robust creative AI systems and highlight considerations around evaluation, scalability, and responsible deployment in creative domains.

Abstract

Large Language Models (LLMs) often struggle with creative generation, and multi-agent frameworks that improve reasoning through interaction can paradoxically hinder creativity by inducing content homogenization. We introduce LLM Review, a peer-review-inspired framework implementing Blind Peer Review: agents exchange targeted feedback while revising independently, preserving divergent creative trajectories. To enable rigorous evaluation, we propose SciFi-100, a science fiction writing dataset with a unified framework combining LLM-as-a-judge scoring, human annotation, and rule-based novelty metrics. Experiments demonstrate that LLM Review consistently outperforms multi-agent baselines, and smaller models with our framework can surpass larger single-agent models, suggesting interaction structure may substitute for model scale.

Figures (6)

• Figure 1: Comparison of multi-agent framework, from single-agent zero-shot writing to multi-agent frameworks. A single LLM generates a story in one pass without feedback, while LLM Teacher, LLM Debate, and LLM Discussion introduce hierarchical guidance, discussion, or role-based collaboration. LLM Review (ours) adopts a blind peer-review topology that decouples critique from generation, enabling independent revisions and reducing homogenization.
• Figure 2: The average score of 5 LLM-as-a-judge evaluation aspects with different Top-p decoding methods.
• Figure 3: The average score of 5 LLM-as-a-judge evaluation aspects with different temperatures.
• Figure 4: Number of execution rounds vs the average score of 5 LLM-as-a-judge evaluation aspects.
• Figure 5: Number of agents vs the average score of 5 LLM-as-a-judge evaluation aspects.