Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Taxonomy of Prompt Modifiers for Text-To-Image Generation

Jonas Oppenlaender

TL;DR

The paper tackles the lack of structured understanding of how practitioners craft prompts for text-to-image generation by presenting a six-type taxonomy of prompt modifiers derived from a three-month autoethnographic and online ethnographic study. It details how prompt modifiers are applied, including iterative workflows, weighting, and the nuanced roles of different modifier categories in shaping output quality and style. The work contributes a principled framework for studying prompt engineering within HCI, highlighting social dynamics, co-creation, bias, and computational aesthetics as key research directions. Its findings have practical implications for tooling, education, and responsible deployment of generative image systems in creative contexts.

Abstract

Text-to-image generation has seen an explosion of interest since 2021. Today, beautiful and intriguing digital images and artworks can be synthesized from textual inputs ("prompts") with deep generative models. Online communities around text-to-image generation and AI generated art have quickly emerged. This paper identifies six types of prompt modifiers used by practitioners in the online community based on a 3-month ethnographic study. The novel taxonomy of prompt modifiers provides researchers a conceptual starting point for investigating the practice of text-to-image generation, but may also help practitioners of AI generated art improve their images. We further outline how prompt modifiers are applied in the practice of "prompt engineering." We discuss research opportunities of this novel creative practice in the field of Human-Computer Interaction (HCI). The paper concludes with a discussion of broader implications of prompt engineering from the perspective of Human-AI Interaction (HAI) in future applications beyond the use case of text-to-image generation and AI generated art.

A Taxonomy of Prompt Modifiers for Text-To-Image Generation

TL;DR

The paper tackles the lack of structured understanding of how practitioners craft prompts for text-to-image generation by presenting a six-type taxonomy of prompt modifiers derived from a three-month autoethnographic and online ethnographic study. It details how prompt modifiers are applied, including iterative workflows, weighting, and the nuanced roles of different modifier categories in shaping output quality and style. The work contributes a principled framework for studying prompt engineering within HCI, highlighting social dynamics, co-creation, bias, and computational aesthetics as key research directions. Its findings have practical implications for tooling, education, and responsible deployment of generative image systems in creative contexts.

Abstract

Text-to-image generation has seen an explosion of interest since 2021. Today, beautiful and intriguing digital images and artworks can be synthesized from textual inputs ("prompts") with deep generative models. Online communities around text-to-image generation and AI generated art have quickly emerged. This paper identifies six types of prompt modifiers used by practitioners in the online community based on a 3-month ethnographic study. The novel taxonomy of prompt modifiers provides researchers a conceptual starting point for investigating the practice of text-to-image generation, but may also help practitioners of AI generated art improve their images. We further outline how prompt modifiers are applied in the practice of "prompt engineering." We discuss research opportunities of this novel creative practice in the field of Human-Computer Interaction (HCI). The paper concludes with a discussion of broader implications of prompt engineering from the perspective of Human-AI Interaction (HAI) in future applications beyond the use case of text-to-image generation and AI generated art.
Paper Structure (23 sections, 3 figures, 2 tables)

This paper contains 23 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Text-to-Image Generation
  4. Prompt Engineering and Prompt Modifiers
  5. Method
  6. Autoethnographic Research on Prompt Engineering
  7. Ethnographic Study of the Text-to-Image Art Community
  8. Twitter community
  9. Review of community resources
  10. Inductive Development of the Taxonomy
  11. Self-Disclosure
  12. Taxonomy of Prompt Modifiers
  13. Prompt Engineering in Practice
  14. Discussion
  15. Broader Implications for Human-AI Interaction
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Selected images generated with text-to-image generation using VQGAN--CLIP (top), Midjourney.com (middle), and DALL-E 2 (bottom).
  • Figure 2: Digital artwork generated with DISCO Diffusion from the input prompt "A beautiful painting of a singular lighthouse, shining its light across a tumultuous sea of blood by greg rutkowski and thomas kinkade, Trending on artstation." This prompt is part of the default configuration settings in the DISCO Diffusion notebook.
  • Figure 3: Example of iterative prompt engineering for generating an image. Images generated with VQGAN--CLIP by VQGANCLIP with 175 iterations, CLIP model ViT-B/32, VQGAN model wikiart_16384, and seed 6087304447281500163.