Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inspo: Writing with Crowds Alongside AI

Chieh-Yang Huang, Sanjana Gautam, Shannon McClellan Brooks, Ya-Fang Lin, Tiffany Knearem, Ting-Hao 'Kenneth' Huang

TL;DR

How crowd-writing systems, in the large language model (LLM) era, can shift to fostering LLM-human collaboration is discussed, with participants favoring AI due to its faster responses and more consistent quality.

Abstract

The use of artificial intelligence (AI) to support creative writing has bloomed in recent years. However, it is less well understood how AI compares to on-demand human support. We explored how writers interact with both AI and crowd worker writing assistants in creative writing. We replicated the interface of the prior crowd-writing system, Heteroglossia, and developed Inspo, a text editor allowing users to request suggestions from AI models and crowd workers. In a one-week deployment study involving eight creative writers, we examined how often participants selected crowd workers when fluent AI text generators were also available. Findings showed a consistent decline in crowd worker usage, with participants favoring AI due to its faster responses and more consistent quality. We conclude with suggestions for future systems, recommending designs that account for the unique strengths and weaknesses of human versus AI assistants, strategies to address automation bias, and sociocultural views of writing.

Inspo: Writing with Crowds Alongside AI

TL;DR

How crowd-writing systems, in the large language model (LLM) era, can shift to fostering LLM-human collaboration is discussed, with participants favoring AI due to its faster responses and more consistent quality.

Abstract

The use of artificial intelligence (AI) to support creative writing has bloomed in recent years. However, it is less well understood how AI compares to on-demand human support. We explored how writers interact with both AI and crowd worker writing assistants in creative writing. We replicated the interface of the prior crowd-writing system, Heteroglossia, and developed Inspo, a text editor allowing users to request suggestions from AI models and crowd workers. In a one-week deployment study involving eight creative writers, we examined how often participants selected crowd workers when fluent AI text generators were also available. Findings showed a consistent decline in crowd worker usage, with participants favoring AI due to its faster responses and more consistent quality. We conclude with suggestions for future systems, recommending designs that account for the unique strengths and weaknesses of human versus AI assistants, strategies to address automation bias, and sociocultural views of writing.
Paper Structure (19 sections, 4 figures, 2 tables)

This paper contains 19 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction and Backgrounds
  2. Inspo System
  3. Four Ways of Generating Suggestions
  4. User Study
  5. Participants.
  6. Writing Task.
  7. Two-Part User Study.
  8. Analyses and Findings
  9. RQ1: How frequently is each suggestion generation method used?
  10. GPT3-Continuation was the most popular, while Crowd usage declined after the initial phase.
  11. Crowd had a much longer system and reading latency, with one-third of responses going unread.
  12. RQ2: How does each method of suggestion affect word count?
  13. GPT3-Continuation boosted word production the most; Crowd might make writers think longer.
  14. RQ3: How does each suggestion method influence writing content?
  15. GPT3-Continuation influenced the written content the most, with full-sentence adoption across all methods.
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Inspo contains (A) a text editor, (B) a suggestion panel, and (C) a sidebar. Users can select any part of a working draft and send it to Inspo to request suggestions from either the text-generation model of their choice (including GPT-3) or online crowd workers.
  • Figure 2: System and reading latency (in seconds) for four different tasks (RQ1). C: Crowd, S: Story Plot Generator, GP: GPT3-Plot, and GC: GPT3-Continuation. Note that P002 $_{na}$ is excluded. For Crowd, half took up to 60 minutes for an initial response, and users typically read the response hours later. A higher rate of not reading is also observed.
  • Figure 3: Distribution of the writing progress (RQ2). In (a), GPT3-Continuation shows a higher density near 100 words; Crowd comes with a small peak at 70 words. This peak reduces and shifts left in (b), indicating that users might need time to catch up with the prior information. (We set common_norm to false to ensure each distribution's area sums to one. We also clipped values to $[0, 200]$.)
  • Figure 4: Distribution of the similarity metrics (RQ3). A 1.0 similarity means that the whole sentence is adopted. Overall, GPT3-Continuation has a higher rate of being adopted.