Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks

Jennifer Haase, Jana Gonnermann-Müller, Paul H. P. Hanel, Nicolas Leins, Thomas Kosch, Jan Mendling, Sebastian Pokutta

TL;DR

This study decomposes variance in open-ended LLM creative outputs into three sources: prompts, between-model differences, and within-LLM sampling fluctuations. By applying a linear mixed-effects framework to 12 LLMs across 10 prompts with 100 samples each on the Alternate Uses Task, it reveals that originality is almost evenly split between prompts and model choice, with substantial but smaller within-LLM variance; fluency is dominated by model differences and sampling variability. The work highlights that prompts act as distributions shapers rather than deterministic editors, and that high-quality prompts can incur greater variability across models, necessitating multiple samples for robust benchmarking. Practically, this implies that single-sample evaluations can misattribute effects, and future creativity benchmarks should report variance components and sample complexity to enable fair, reproducible comparisons across systems.

Abstract

How much of LLM output variance is explained by prompts versus model choice versus stochasticity through sampling? We answer this by evaluating 12 LLMs on 10 creativity prompts with 100 samples each (N = 12,000). For output quality (originality), prompts explain 36.43% of variance, comparable to model choice (40.94%). But for output quantity (fluency), model choice (51.25%) and within-LLM variance (33.70%) dominate, with prompts explaining only 4.22%. Prompts are powerful levers for steering output quality, but given the substantial within-LLM variance (10-34%), single-sample evaluations risk conflating sampling noise with genuine prompt or model effects.

Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks

TL;DR

This study decomposes variance in open-ended LLM creative outputs into three sources: prompts, between-model differences, and within-LLM sampling fluctuations. By applying a linear mixed-effects framework to 12 LLMs across 10 prompts with 100 samples each on the Alternate Uses Task, it reveals that originality is almost evenly split between prompts and model choice, with substantial but smaller within-LLM variance; fluency is dominated by model differences and sampling variability. The work highlights that prompts act as distributions shapers rather than deterministic editors, and that high-quality prompts can incur greater variability across models, necessitating multiple samples for robust benchmarking. Practically, this implies that single-sample evaluations can misattribute effects, and future creativity benchmarks should report variance components and sample complexity to enable fair, reproducible comparisons across systems.

Abstract

How much of LLM output variance is explained by prompts versus model choice versus stochasticity through sampling? We answer this by evaluating 12 LLMs on 10 creativity prompts with 100 samples each (N = 12,000). For output quality (originality), prompts explain 36.43% of variance, comparable to model choice (40.94%). But for output quantity (fluency), model choice (51.25%) and within-LLM variance (33.70%) dominate, with prompts explaining only 4.22%. Prompts are powerful levers for steering output quality, but given the substantial within-LLM variance (10-34%), single-sample evaluations risk conflating sampling noise with genuine prompt or model effects.
Paper Structure (34 sections, 1 equation, 11 figures, 6 tables)

This paper contains 34 sections, 1 equation, 11 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt Engineering and Benchmarkings
  4. Stochasticity and Inference Variability
  5. Methodological Gap: The Sampling Fallacy
  6. Methodology
  7. Stimuli: Prompt Design
  8. Inference Configuration
  9. Procedure and Data Collection
  10. Evaluation & Analysis
  11. Data Preprocessing.
  12. Variance Decomposition.
  13. Content Analysis (Thematic & Structural).
  14. Results
  15. Variance Decomposition
  16. ...and 19 more sections

Figures (11)

  • Figure 1: Variance Decomposition (ICC) for Originality and Fluency.
  • Figure 2: Violinplots displaying originality scores as a function of prompt. Prompt Key: P1: Direct Baseline (Think of uses...). P2: One-Shot Example (Example: Use for yarn storage...). P3: Heuristic/Domain (Think across domains: art, survival...). P4: Anticipatory (Avoid generic ideas...). P5: Chain-of-Thought (Think step-by-step...). P6: Creative Persona (You are the most creative person...). P7: Phrasing Variation (Synonymous rewording). P8: Format Constraint (No titles or colons...). P9: Info Order (Emphasize inventive...). P10: Typo Robustness (Injected noise).
  • Figure 3: Heatmap of Mean Originality Scores (Model $\times$ Prompt). Prompt key as in Figure \ref{['fig:prompt_originality_by_prompt_violin']}.
  • Figure 4: Quality vs. Quantity Trade-off. Models cluster into "Reasoning-Oriented" (top-left: high originality, low fluency) and "High-Fluency" (middle-right: high volume, moderate quality) architectures.
  • Figure 5: Mean Originality by Model (95% CI).
  • ...and 6 more figures