Partnering with Generative AI: Experimental Evaluation of Human-Led and Model-Led Interaction in Human-AI Co-Creation

TL;DR

This paper investigates how the mode of human-LLM interaction influences co-creation outcomes in creativity tasks. Through a preregistered randomized experiment with five conditions, it shows that model-led refinement boosts idea quality but hurts diversity and ownership, while a human-led question-driven mode enhances quality, preserves ownership, and expands idea diversity. The findings highlight the value of reflective, human-centric collaboration and offer design guidelines to balance AI capability with human creative agency. Practically, the work advocates embedding question-driven interactions as a standard pattern to mitigate homogenization and sustain user engagement in AI-assisted ideation.

Abstract

Large language models (LLMs) show strong potential to support creative tasks, but the role of the interface design is poorly understood. In particular, the effect of different modes of collaboration between humans and LLMs on co-creation outcomes is unclear. To test this, we conducted a randomized controlled experiment ($N = 486$) comparing: (a) two variants of reflective, human-led modes in which the LLM elicits elaboration through suggestions or questions, against (b) a proactive, model-led mode in which the LLM independently rewrites ideas. By assessing the effects on idea quality, diversity, and perceived ownership, we found that the model-led mode substantially improved idea quality but reduced idea diversity and users' perceived idea ownership. The reflective, human-led mode also improved idea quality, yet while preserving diversity and ownership. Our findings highlight the importance of designing interactions with generative AI systems as reflective thought partners that complement human strengths and augment creative processes.

Figures (7)

• Figure 1: Overview of the study procedure.(A) Participants ($N=486$) were randomly assigned to one of five conditions: four LLM variants (vanilla, model-led, suggestion, question) or a control group without support. (B) Before submitting their final ideas, all participants were instructed to refine their idea using the LLM. (C) Finally, idea quality was assessed by expert raters with domain expertise in the automotive industry; idea diversity was measured via text-embedding similarity of idea descriptions; and perceived ownership was self-reported by participants for the ideas they generated.
• Figure 2: Screenshots of the interface of the two human-led conditions. In the question-mode, the LLM provided (A) further knowledge and (B) an analogy to inspire human creativity. After defining three alternative ideas, the user chooses one to elaborate further. In the suggestion-mode, the participants were provided three suggestions on how to make the initial idea more novel, and on how to further elaborate (C). Below the suggestion, a text box is shown where the participants can refine their initial idea based on the LLM input (D).
• Figure 3: Distribution of idea diversity, idea quality, and perceived idea ownership across experimental conditions. Violin plots show the probability density of each measure, with wider sections indicating more frequent values. The boxplots show the interquartile ranges as well as the min/max, and the white dots indicate the medians.
• Figure 4: Distribution of perceived cognitive workload across experimental conditions. Violin plots show the probability density of each measure, with wider sections indicating more frequent values. White dots indicate medians, thick black bars show interquartile ranges.
• Figure 5: Within-subject analysis.(A) We compare the idea diversity of the initial ideas without interaction with the LLM with (B) the idea diversity of the refined ideas after interacting with the LLM