Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Prompt Engineering Practices in the Enterprise

Michael Desmond, Michelle Brachman

TL;DR

It is hypothesized that the way in which users iterate on their prompts can provide insight into how they think prompting and models work, as well as the kinds of support needed for more efficient prompt engineering.

Abstract

Interaction with Large Language Models (LLMs) is primarily carried out via prompting. A prompt is a natural language instruction designed to elicit certain behaviour or output from a model. In theory, natural language prompts enable non-experts to interact with and leverage LLMs. However, for complex tasks and tasks with specific requirements, prompt design is not trivial. Creating effective prompts requires skill and knowledge, as well as significant iteration in order to determine model behavior, and guide the model to accomplish a particular goal. We hypothesize that the way in which users iterate on their prompts can provide insight into how they think prompting and models work, as well as the kinds of support needed for more efficient prompt engineering. To better understand prompt engineering practices, we analyzed sessions of prompt editing behavior, categorizing the parts of prompts users iterated on and the types of changes they made. We discuss design implications and future directions based on these prompt engineering practices.

Exploring Prompt Engineering Practices in the Enterprise

TL;DR

It is hypothesized that the way in which users iterate on their prompts can provide insight into how they think prompting and models work, as well as the kinds of support needed for more efficient prompt engineering.

Abstract

Interaction with Large Language Models (LLMs) is primarily carried out via prompting. A prompt is a natural language instruction designed to elicit certain behaviour or output from a model. In theory, natural language prompts enable non-experts to interact with and leverage LLMs. However, for complex tasks and tasks with specific requirements, prompt design is not trivial. Creating effective prompts requires skill and knowledge, as well as significant iteration in order to determine model behavior, and guide the model to accomplish a particular goal. We hypothesize that the way in which users iterate on their prompts can provide insight into how they think prompting and models work, as well as the kinds of support needed for more efficient prompt engineering. To better understand prompt engineering practices, we analyzed sessions of prompt editing behavior, categorizing the parts of prompts users iterated on and the types of changes they made. We discuss design implications and future directions based on these prompt engineering practices.
Paper Structure (22 sections, 6 figures, 2 tables)

This paper contains 22 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompting Practices
  4. Mental Models and Repair
  5. Methods
  6. Data Collection
  7. Qualitative Analysis
  8. Results
  9. Prompt Engineering Sessions: High-level Editing Analysis
  10. Prompt Editing Practices
  11. Use cases
  12. Frequency of Prompt Component and Edit Types
  13. Multiple Edits
  14. Rollbacks
  15. Context
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Duration of observed prompt editing sessions in minutes.
  • Figure 2: The size of change between successive prompts, represented as a similarity ratio. Values closer to 1 indicate similarity between successive prompts, 1 being an exact match, while a ratio of 0 indicates nothing in common.
  • Figure 3: The occurrence of parameter changes as a percentage of sessions in which the change was observed. Users primarily changed the target language model, the maximum number of tokens to generate, and repetition penalty. Stop sequence, temperature and decoding method were also commonly changed.
  • Figure 4: The number of models used per session.
  • Figure 5: Number of edits that focused on each of the prompt components. Users primarily edited context, and task instructions to a lesser extent.
  • ...and 1 more figures