Can Good Writing Be Generative? Expert-Level AI Writing Emerges through Fine-Tuning on High-Quality Books

TL;DR

The study experimentally tests whether high-quality writing can be generated by AI by pitting 28 MFA writers against three LLMs in emulating 50 acclaimed authors across two AI configurations: in-context prompting and fine-tuning on complete author works. Results show a split: experts favor human writing under in-context prompts, but fine-tuning shifts preferences toward AI for both expert and lay judges, while lay judges consistently prefer AI regardless of prompting. Debrief interviews reveal deep identity disruptions among writers, including erosion of aesthetic confidence and a redefinition of writing’s purpose toward process and intention. The findings challenge assumptions about AI’s creative limits, highlight potential labor-market dilution through style extraction, and raise policy questions about disclosure, copyright, and the structure of creative writing education and publishing. Collectively, the work suggests that as AI approaches human-like mastery of authorial voice, the literary ecosystem will need new norms, protections, and training to navigate evolving creative labor dynamics.

Abstract

Creative writing has long been considered a uniquely human endeavor, requiring voice and style that machines could not replicate. This assumption is challenged by Generative AI that can emulate thousands of author styles in seconds with negligible marginal labor. To understand this better, we conducted a behavioral experiment where 28 MFA writers (experts) competed against three LLMs in emulating 50 critically acclaimed authors. Based on blind pairwise comparisons by 28 expert judges and 131 lay judges, we find that experts preferred human writing in 82.7% of cases under the in-context prompting condition but this reversed to 62% preference for AI after fine-tuning on authors' complete works. Lay judges, however, consistently preferred AI writing. Debrief interviews with expert writers revealed that their preference for AI writing triggered an identity crisis, eroding aesthetic confidence and questioning what constitutes "good writing." These findings challenge discourse about AI's creative limitations and raise fundamental questions about the future of creative labor.

Figures (13)

• Figure 1: Three-phase study design showing (Phase 1) writing task with In-context prompting and fine-tuning approaches using authors with critically acclaimed style/voice. In the figure both writers and AI are trying to write a specific excerpt from My Year of Rest and Relaxation in the style/voice of Ottessa Moshfegh (Phase 2) Human-written vs AI-generated text evaluation task with rationales, and (Phase 3) Debrief session exploring sentiments and sensemaking when writers prefer AI over human writing. Figure illustrated by the first author of the paper
• Figure 2: Writing Prompt to emulate Ottessa Moshfegh's style/voice
• Figure 3: Segmented excerpt and extracted content from the excerpt
• Figure 4: The pipeline used to fine-tune GPT-4o on an author's entire oeuvre.
• Figure 5: Number of tokens used for in-context prompting.