Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation

Orit Shaer, Angelora Cooper, Osnat Mokryn, Andrew L. Kun, Hagit Ben Shoshan

TL;DR

This paper devised a collaborative group-AI Brainwriting ideation framework, which incorporated an LLM as an enhancement into the group ideation process, and evaluated the idea generation process and the resulted solution space.

Abstract

The growing availability of generative AI technologies such as large language models (LLMs) has significant implications for creative work. This paper explores twofold aspects of integrating LLMs into the creative process - the divergence stage of idea generation, and the convergence stage of evaluation and selection of ideas. We devised a collaborative group-AI Brainwriting ideation framework, which incorporated an LLM as an enhancement into the group ideation process, and evaluated the idea generation process and the resulted solution space. To assess the potential of using LLMs in the idea evaluation process, we design an evaluation engine and compared it to idea ratings assigned by three expert and six novice evaluators. Our findings suggest that integrating LLM in Brainwriting could enhance both the ideation process and its outcome. We also provide evidence that LLMs can support idea evaluation. We conclude by discussing implications for HCI education and practice.

AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation

TL;DR

This paper devised a collaborative group-AI Brainwriting ideation framework, which incorporated an LLM as an enhancement into the group ideation process, and evaluated the idea generation process and the resulted solution space.

Abstract

The growing availability of generative AI technologies such as large language models (LLMs) has significant implications for creative work. This paper explores twofold aspects of integrating LLMs into the creative process - the divergence stage of idea generation, and the convergence stage of evaluation and selection of ideas. We devised a collaborative group-AI Brainwriting ideation framework, which incorporated an LLM as an enhancement into the group ideation process, and evaluated the idea generation process and the resulted solution space. To assess the potential of using LLMs in the idea evaluation process, we design an evaluation engine and compared it to idea ratings assigned by three expert and six novice evaluators. Our findings suggest that integrating LLM in Brainwriting could enhance both the ideation process and its outcome. We also provide evidence that LLMs can support idea evaluation. We conclude by discussing implications for HCI education and practice.
Paper Structure (42 sections, 1 equation, 5 figures, 4 tables)

This paper contains 42 sections, 1 equation, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Structured Approaches to Ideation
  4. Human-AI Co-Creation
  5. Approaches for Evaluating Ideas
  6. Collaborative Group-AI Brainwriting Framework Design
  7. Brainwriting Divergence stage
  8. Phase 1: Brainwriting using Conceptboard
  9. Phase 2: Enhancing Ideas with an LLM
  10. Brainwriting convergence stage
  11. Phase 3: Selecting and developing ideas through discussion
  12. Can LLMs Help with Convergence? Developing and implementing an LLM Powered Evaluation Engine
  13. Creating a per scale value and criterion definition.
  14. Implementation
  15. User Study: Collaborative Group-AI Brainwriting Process
  16. ...and 27 more sections

Figures (5)

  • Figure 1: Collaborative Group-AI Brainwriting Process
  • Figure 2: The three main areas of Conceptboards used by teams during the Brainwriting session.
  • Figure 3: Idea evaluation with GPT-4 using the proposed scales for relevance, innovation, and insightfulness
  • Figure 4: Identifying biases in LLM-generated ideas. (a) introduces the top terms used in all ideas generated either by humans or by GPT-3, as calculated using the Latent Personal Analysis (LPA) method. (b) depicts GPT-3's LPA signature, denoting its unique use of terms when compared to the shared vocabulary, either underused or overused.
  • Figure 5: The Distribution of ratings on a 1 to 5 Likert scale given to ideas generated in the Brainwriting process. Ideas were generated by either humans, GPT-3, or as a collaboration. Every idea was assessed based on three criteria: its relevance, depth of insight, and level of innovation. All 148 ideas were rated by Experts, Novices, and the GPT-4 rating engine. The lower panel depicts the distribution of ratings given by Experts to ideas in each of the criteria. The middle panel depicts ratings given by Novices, and the upper panel the rates given by GPT-4.