Prompt Stability Matters: Evaluating and Optimizing Auto-Generated Prompt in General-Purpose Systems

TL;DR

Prompt Stability Matters addresses the unreliability of automatic prompt generation in general-purpose, LLM-based multi-agent systems due to intrinsic stochasticity. It defines semantic stability as an embedding-based measure of output consistency and builds a stability-aware framework, Promptor, that uses stability feedback to refine prompts and improve system-wide task success. A planner–executor analysis provides a probabilistic bound linking execution deviation to per-agent variance, justifying stability as a necessary condition for reliability. Empirically, Promptor yields higher accuracy and more consistent outputs across diverse general and domain-specific tasks, demonstrating practical benefits for trustworthy, scalable multi-agent LLM systems.

Abstract

Automatic prompt generation plays a crucial role in enabling general-purpose multi-agent systems to perform diverse tasks autonomously. Existing methods typically evaluate prompts based on their immediate task performance, overlooking the intrinsic qualities that determine their reliability. This outcome-centric view not only limits interpretability but also fails to account for the inherent stochasticity of large language models (LLMs). In this work, we bring attention to prompt stability-the consistency of model responses across repeated executions-as a key factor for building robust and effective prompt generation systems. To quantify this, we propose semantic stability as a criterion for assessing the response consistency of prompts, and fine-tune a LLaMA-based evaluator to measure it automatically across tasks. These components have enabled us to develop the first stability-aware general-purpose prompt generation system that leverages stability feedback to iteratively enhance both prompt quality and system-level performance. Furthermore, we establish a logical chain between prompt stability and task success by analyzing the structural dependencies within our system, proving stability as a necessary condition for effective system-level execution. Empirical results across general and domain-specific tasks demonstrate that our stability-aware framework improves both accuracy and output consistency. By shifting the focus from one-off results to persistent reliability, our work offers a new perspective on prompt design and contributes practical tools for building more trustworthy general-purpose systems.

Figures (4)

• Figure 1: Promptor system pipeline of our stability-aware prompt generation framework.
• Figure 2: Illustration of semantic stability as a metric.
• Figure 3: Performance on MATH
• Figure 4: Performance on ML-Benchmark

Theorems & Definitions (1)

• Lemma 1