Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Survey of User Interface Design and Interaction Techniques in Generative AI Applications

Reuben Luera, Ryan A. Rossi, Alexa Siu, Franck Dernoncourt, Tong Yu, Sungchul Kim, Ruiyi Zhang, Xiang Chen, Hanieh Salehy, Jian Zhao, Samyadeep Basu, Puneet Mathur, Nedim Lipka

TL;DR

A survey that comprehensively presents taxonomies of how a human interacts with AI and the user interaction patterns designed to meet the needs of a variety of relevant use cases is presented.

Abstract

The applications of generative AI have become extremely impressive, and the interplay between users and AI is even more so. Current human-AI interaction literature has taken a broad look at how humans interact with generative AI, but it lacks specificity regarding the user interface designs and patterns used to create these applications. Therefore, we present a survey that comprehensively presents taxonomies of how a human interacts with AI and the user interaction patterns designed to meet the needs of a variety of relevant use cases. We focus primarily on user-guided interactions, surveying interactions that are initiated by the user and do not include any implicit signals given by the user. With this survey, we aim to create a compendium of different user-interaction patterns that can be used as a reference for designers and developers alike. In doing so, we also strive to lower the entry barrier for those attempting to learn more about the design of generative AI applications.

Survey of User Interface Design and Interaction Techniques in Generative AI Applications

TL;DR

A survey that comprehensively presents taxonomies of how a human interacts with AI and the user interaction patterns designed to meet the needs of a variety of relevant use cases is presented.

Abstract

The applications of generative AI have become extremely impressive, and the interplay between users and AI is even more so. Current human-AI interaction literature has taken a broad look at how humans interact with generative AI, but it lacks specificity regarding the user interface designs and patterns used to create these applications. Therefore, we present a survey that comprehensively presents taxonomies of how a human interacts with AI and the user interaction patterns designed to meet the needs of a variety of relevant use cases. We focus primarily on user-guided interactions, surveying interactions that are initiated by the user and do not include any implicit signals given by the user. With this survey, we aim to create a compendium of different user-interaction patterns that can be used as a reference for designers and developers alike. In doing so, we also strive to lower the entry barrier for those attempting to learn more about the design of generative AI applications.

Paper Structure

This paper contains 59 sections, 14 figures, 1 table.

Table of Contents

  1. Introduction
  2. Summary of Main Contributions
  3. Scope of this Article
  4. Background & Preliminaries
  5. User-Guided Interactions
  6. Prompts and Inputs
  7. Input Modalities
  8. Text-based
  9. Visual
  10. Sound
  11. User-Guided Interactions
  12. Prompting
  13. Text-based Prompts
  14. Visual Prompts
  15. Audio Prompts
  16. ...and 44 more sections

Figures (14)

  • Figure 1: Prompt vs Inputs (Sec. \ref{['sec:input-modalities']}): A visual summary of the distinction between prompts and inputs. A prompt is a user-guided interaction where the user asks the system to complete a task. Whereas the input is the piece of data, information, or content that the prompt is acting upon.
  • Figure 2: Modalities: A high-level visual summary of the different modalities that generative AIs use (Sec. \ref{['sec:input-modalities']}).
  • Figure 3: Taxonomy of works by their input/output modalities.
  • Figure 4: User-Guided Interaction Taxonomy. Generative AI systems and tools are summarized using the proposed user-guided interaction taxonomy (Sec. \ref{['sec:userguided']}).
  • Figure 5: Prompting Visual Summary (Sec. \ref{['sec:prompting']}): An overview of the four main prompting subcategories. Prompting is a user-guided interaction where a user asks or "prompts" the generative AI system to complete a certain task.
  • ...and 9 more figures

Theorems & Definitions (3)

  • Definition 1: User Guided Interaction
  • Definition 2: Inputs
  • Definition 3: Prompts