Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt Stability Scoring for Text Annotation with Large Language Models

Christopher Barrie, Elli Palaiologou, Petter Törnberg

TL;DR

This paper tackles the challenge that small prompt changes can undermine the reproducibility of zero-/few-shot LM text annotation. It introduces the Prompt Stability Score (PSS) and a Python package to diagnose both intra-prompt and inter-prompt stability by re-running the same prompt and by generating semantically similar prompt variants, respectively. Through six social-science datasets and twelve outcomes, the authors demonstrate that intra-prompt stability is generally high but inter-prompt stability declines with semantic drift and task difficulty, and they discuss cost, prompt reliability, and cross-model considerations. The work provides a practical, model-agnostic toolkit and guidance to evaluate prompt reliability before validation, aiming to reduce replication risk in LM-based classification tasks.

Abstract

Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call ``prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package \texttt{promptstability} for its estimation. Using six different datasets and twelve outcomes, we classify $\sim$3.1m rows of data and $\sim$300m input tokens to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.

Prompt Stability Scoring for Text Annotation with Large Language Models

TL;DR

This paper tackles the challenge that small prompt changes can undermine the reproducibility of zero-/few-shot LM text annotation. It introduces the Prompt Stability Score (PSS) and a Python package to diagnose both intra-prompt and inter-prompt stability by re-running the same prompt and by generating semantically similar prompt variants, respectively. Through six social-science datasets and twelve outcomes, the authors demonstrate that intra-prompt stability is generally high but inter-prompt stability declines with semantic drift and task difficulty, and they discuss cost, prompt reliability, and cross-model considerations. The work provides a practical, model-agnostic toolkit and guidance to evaluate prompt reliability before validation, aiming to reduce replication risk in LM-based classification tasks.

Abstract

Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call ``prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package \texttt{promptstability} for its estimation. Using six different datasets and twelve outcomes, we classify 3.1m rows of data and 300m input tokens to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.
Paper Structure (25 sections, 10 equations, 9 figures, 6 tables)

This paper contains 25 sections, 10 equations, 9 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Coder reliability and prompt stability
  3. Current Approaches
  4. Design
  5. Data
  6. Method
  7. Intra-prompt stability
  8. Inter-prompt stability
  9. Results
  10. How much does this all cost?
  11. Are the generated prompts actually reliable?
  12. Are other LMs more stable?
  13. What constitutes acceptable stability?
  14. What about inter-model stability?
  15. Parameter stability
  16. ...and 10 more sections

Figures (9)

  • Figure 1: Combined intra-prompt stability scores ordered by average score.
  • Figure 2: Combined inter-prompt stability scores, ordered by average score.
  • Figure 3: Combined inter-prompt stability scores for random subsamples.
  • Figure 4: Combined inter-prompt stability scores for lowest inter-PSS outcomes in main analyses.
  • Figure 5: Combined intra- and inter-prompt stability scores for DeepSeek-R1 and GPT-4o.
  • ...and 4 more figures