Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Taxonomy of Human--MLLM Interaction in Early-Stage Sketch-Based Design Ideation

Weiayn Shi, Kenny Tsu Wei Choo

TL;DR

Analysis of sketch-based design interactions with an MLLM-powered system using automatically recorded interaction logs and post-task interviews shows that designers rarely rely on a single mode and human-led and AI-led roles are frequently interwoven and shift across ideation instances.

Abstract

As multimodal large language models (MLLMs) are increasingly integrated into early-stage design tools, it is important to understand how designers collaborate with AI during ideation. In a user study with 12 participants, we analysed sketch-based design interactions with an MLLM-powered system using automatically recorded interaction logs and post-task interviews. Based on how creative responsibility was allocated between humans and the AI, we predefined four interaction modes: Human-Only, Human-Lead, AI-Lead, and Co-Evolution, and analysed how these modes manifested during sketch-based design ideation. Our results show that designers rarely rely on a single mode; instead, human-led and AI-led roles are frequently interwoven and shift across ideation instances. These findings provide an empirical basis for future work to investigate why designers shift roles with AI and how interactive systems can better support such dynamic collaboration.

A Taxonomy of Human--MLLM Interaction in Early-Stage Sketch-Based Design Ideation

TL;DR

Analysis of sketch-based design interactions with an MLLM-powered system using automatically recorded interaction logs and post-task interviews shows that designers rarely rely on a single mode and human-led and AI-led roles are frequently interwoven and shift across ideation instances.

Abstract

As multimodal large language models (MLLMs) are increasingly integrated into early-stage design tools, it is important to understand how designers collaborate with AI during ideation. In a user study with 12 participants, we analysed sketch-based design interactions with an MLLM-powered system using automatically recorded interaction logs and post-task interviews. Based on how creative responsibility was allocated between humans and the AI, we predefined four interaction modes: Human-Only, Human-Lead, AI-Lead, and Co-Evolution, and analysed how these modes manifested during sketch-based design ideation. Our results show that designers rarely rely on a single mode; instead, human-led and AI-led roles are frequently interwoven and shift across ideation instances. These findings provide an empirical basis for future work to investigate why designers shift roles with AI and how interactive systems can better support such dynamic collaboration.
Paper Structure (12 sections, 2 figures)

This paper contains 12 sections, 2 figures.

Table of Contents

  1. Introduction
  2. User Study
  3. Study Prototype: SketchLLM
  4. Study Design
  5. Participants
  6. Result
  7. Co-Evolution
  8. AI-Lead
  9. Human-Lead
  10. Human-only
  11. Discussion
  12. Limitations.

Figures (2)

  • Figure 1: Overview of the SketchLLM: (a)Sketching module: Users draw early-stage product concepts (e.g., a toaster) using stylus input on the canvas, which includes a sketch gallery, drawing tools, and controls for launching the Multimodal AI Chatbot. (b) Text- and image-based interaction and generation with the AI for further exploration.
  • Figure 2: (a) Interaction sequences derived from system logs. Each horizontal bar represents one participant, with coloured segments indicating the sequence of actions taken for individual ideation instances. Action sequences are segmented by ideation episode, with all actions counted equally. User Input (Sketch, Type Prompt); AI Output (Generate Text, Generate Image); and Other (Export Sketch, Import AI Image). (b) Proportional distribution of ideation instances across interaction modes for each participant, normalised to 100%. Colours indicate Human-Only, Human-Lead, AI-Lead, and Co-Evolution.