Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Prompts

TL;DR

It is argued that system prompts warrant explicit, user-centred design attention and, focusing on large language models (LLMs), what do system prompts contain, how do end-users perceive them, and what do these perceptions offer for design and governance practice?

Abstract

System prompts - instructions that shape the behaviour of generative AI systems - strongly influence system outputs and users' experiences. They define the model's guidelines, `personality', and guardrails, taking precedence over user inputs. Despite their influence, transparency is limited: system prompts are generally not made public and most platforms instruct models to conceal them, leaving users disconnected from and unaware of a key mechanism guiding and governing their AI interactions. This paper argues that system prompts warrant explicit, user-centred design attention and, focusing on large language models (LLMs), asks: what do system prompts contain, how do end-users perceive them, and what do these perceptions offer for design and governance practice? Our results reveal user perspectives on: the benefits and risks of system prompts; the values they prefer to be associated with prompt-design; their levels of comfort with different types of prompts; and degrees of transparency and user control regarding prompt content. From these findings emerge considerations for how designers can better align system prompt mechanisms with user expectations and preferences over these mechanisms that directly shape how generative AI systems behave.

Figures (18)

• Figure 1: Example of the instruction hierarchy: Each message in a chat can have different levels of privilege according to the instruction hierarchy. The AI should follow the instructions with the highest priority first. 'System messages' are on top of the hierarchy; every following instruction is only followed if it is in alignment. Figure from Wallace et al.(2024)wallace2024instructionhierarchytrainingllms.
• Figure 2: Excerpts from updates to the system prompt of xAI's 'Ask Grok' function over the course of one week, evidencing rapid prompt changes being implemented to deal with AI misaligning with company goals or user preferences.
• Figure 3: Comparison of topic analyses: Stacked bar charts for the total number of documents matching each topics. Match rate chart for proportion of scored document split by source for each topic. Both compare purpose-configured (blue) and multi-purpose system prompt (chunks) (magenta).
• Figure 4: Flowchart of the survey: Participants move through seven stages (from left to right): Introduction, System Prompt Awareness, Topic-Specific Design Preferences, Overall Design Preferences, Transparency, Interference, and Demographics with steps with user input in green and information pages in light gray. Green arrows denote moving from one stage to another.
• Figure 5: Thirteen Design Values: List of often implicated value-sensitive design values and their descriptions, after sadek_guidelines_2024.