Prompt Optimization via Retrieved Reasoning Assets and Multi-Agent Analysis

TL;DR

MA-SAPO presents a score-aware, two-phase framework for prompt optimization that links evaluation signals to interpretable reasoning assets. In training, three specialized agents convert metrics into reusable artifacts that encode explanations, diagnoses, and actionable edits; in testing, a retrieval-augmented Analyzer–Refiner pipeline applies evidence-backed refinements to new prompts. Across HelpSteer1/2 benchmarks and multiple backbones, MA-SAPO outperforms single-pass, RAG, and prior multi-agent baselines while achieving favorable cost and latency, driven by structured reasoning that makes optimizations transparent and controllable. The work demonstrates that turning evaluation into auditable reasoning assets yields more reliable, interpretable improvements, with human evaluations confirming superior reasoning quality and intent preservation. Limitations include dependence on score quality and a sparse retrieval scheme, pointing to future work in robust asset validation and hybrid retrieval strategies.

Abstract

Prompt optimization has emerged as an effective alternative to retraining for improving the performance of Large Language Models (LLMs). However, most existing approaches treat evaluation as a black box, relying solely on numerical scores while offering limited insight into why a prompt succeeds or fails. They also depend heavily on trial-and-error refinements, which are difficult to interpret and control. In this paper, we introduce MA-SAPO, a Multi-Agent framework for Score-Aware Prompt Optimization. Compared to prior methods, MA-SAPO explicitly couples evaluation outcomes with structured reasoning to guide systematic edits. The framework specifically consists of two stages: during the Reasoning Phase, agents collaboratively explain metric scores, diagnose weaknesses, and synthesize targeted refinements that are stored as reusable reasoning assets; during the Test Phase, agents retrieve these assets to analyze optimized prompts and apply only evidence-grounded edits. By turning evaluation signals into interpretable reasoning chains, MA-SAPO produces prompt refinements that are more transparent, auditable, and controllable. Experiments on the HelpSteer1/2 benchmarks demonstrate consistent improvements over single-pass prompting, retrieval-augmented baselines, and prior multi-agent strategies, validating the effectiveness of our approach.

Figures (4)

• Figure 1: Comparative overview of prompt optimization methods. MA-SAPO enhances interpretability by linking evaluation scores to reasoning-driven and evidence-based refinements.
• Figure 2: Overview of the MA-SAPO framework. The training phase constructs reasoning assets $(\mathcal{R}_i = (\mathcal{C}_i, \mathcal{D}_i, \mathcal{E}_i))$ from annotated prompt–response pairs, while the test phase retrieves top-k training examples and their reasoning assets to refine a new test prompt.
• Figure 3: Distribution of directional consistency ratings (n=560). Ratings on a 4-point scale (1 = changed, 4 = preserved). The dashed line marks the mean (3.36) and the shaded band shows one standard deviation (SD = 0.74).
• Figure 4: Case study illustrating MA-SAPO’s multi-agent prompt optimization process. Starting from an initial user prompt and evaluation scores, the framework retrieves reasoning assets, analyzes weaknesses, and iteratively refines the prompt. The regenerated response shows improved clarity, structure, and contextual depth compared to the original.