Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writers
Tuhin Chakrabarty, Vishakh Padmakumar, Faeze Brahman, Smaranda Muresan
TL;DR
This paper investigates how state-of-the-art large language models can support professional writers by grounding a collaborative writing interface in the cognitive process model of writing (planning, translating, reviewing). Through an empirical study with $n=30$ stories written by $17$ emerging writers and analyzed across writer–LLM interactions and post-study surveys, the authors find that LLMs are most helpful for translation and reviewing tasks, while writers often navigate non-linearly across cognitive activities. The study provides a dataset of co-written stories and interaction logs, shows that model drafts are variably retained (less than $35\%$ in many cases) and highlights key weaknesses (repetition, clichés, lack of nuance, and moralizing endings) as well as user desires (tone control, attribution, and better intention inference). The contributions include the first large-scale dataset of professional-writer interactions with an LLM under a theoretically grounded interface, and several directions for future research in model alignment, prompting strategies, and safety trade-offs to better support creative writing. Overall, the work informs the design of Creativity Support Tools that balance authors’ agency with machine assistance and points to practical implications for training future models to better align with writers’ nuanced needs.
Abstract
The development of large language models (LLMs) capable of following instructions and engaging in conversational interactions sparked increased interest in their utilization across various support tools. We investigate the utility of modern LLMs in assisting professional writers via an empirical user study (n=30). The design of our collaborative writing interface is grounded in the cognitive process model of writing that views writing as a goal-oriented thinking process encompassing non-linear cognitive activities: planning, translating, and reviewing. Participants are asked to submit a post-completion survey to provide feedback on the potential and pitfalls of LLMs as writing collaborators. Upon analyzing the writer-LLM interactions, we find that while writers seek LLM's help across all three types of cognitive activities, they find LLMs more helpful in translation and reviewing. Our findings from analyzing both the interactions and the survey responses highlight future research directions in creative writing assistance using LLMs.