CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities

TL;DR

CoAuthor presents a principled approach to studying language model capabilities in human-AI collaborative writing by designing and releasing a large, richly annotated interaction dataset. The dataset captures 1445 writing sessions between 63 writers and four GPT-3 instances across two task types (creative and argumentative), recording both process and outcome through detailed event blocks and post-session surveys. Analyses reveal GPT-3's language fluency, its potential for ideation via new named entities, and variable collaboration with writers, contingent on individual writers and, to a lesser extent, prompts and randomness. The work argues that such datasets can serve as boundary objects between HCI and NLP, enabling hypothesis formulation, plausibility assessment, and LM training/evaluation, and provides replay tools to foster principled interaction-design discussions and future research directions.

Abstract

Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs' generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3's capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities, and reveal its contribution as a writing "collaborator" under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs' promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.

Figures (13)

• Figure 1: We present CoAuthor, a dataset designed for revealing GPT-3's generative capabilities for interactive writing. It contains rich interactions between 63 writers and 4 instances of GPT-3 across 1445 writing sessions. Each session starts with a prompt (black text). Writers then freely write (brown), request suggestions from GPT-3 (blue), accept or dismiss suggestions, and edit accepted suggestions or previous texts in any order they choose.
• Figure 2: We contrast the capabilities of GPT-3 with high randomness and low randomness in creative and argumentative writing. To control randomness, we varied two decoding parameters: temperature (T) and frequency penalty (FP).
• Figure 3: Interface used for data collection of CoAuthor. Our interface was a text editor in which writers press the tab key to get suggestions from the system whenever desired.
• Figure 4: Sentences written by both writers and GPT-3 had fewer spelling and grammatical errors (a) and contained more diverse vocabulary (b) compared to sentences written by writers alone and GPT-3 alone.
• Figure 5: For both creative (a) and argumentative (b) writing, the number of spelling and grammar errors per word (y-axis) in GPT-3-generated sentences vary across writing prompts (x-axis).