Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses

Minki Hong, Eunsoo Lee, Sohyun Park, Jihie Kim

TL;DR

Overall, PEEM provides a reproducible, criterion-driven protocol that links prompt formulation to response behavior and enables systematic diagnosis and optimization of LLM interactions.

Abstract

Prompt design is a primary control interface for large language models (LLMs), yet standard evaluations largely reduce performance to answer correctness, obscuring why a prompt succeeds or fails and providing little actionable guidance. We propose PEEM (Prompt Engineering Evaluation Metrics), a unified framework for joint and interpretable evaluation of both prompts and responses. PEEM defines a structured rubric with 9 axes: 3 prompt criteria (clarity/structure, linguistic quality, fairness) and 6 response criteria (accuracy, coherence, relevance, objectivity, clarity, conciseness), and uses an LLM-based evaluator to output (i) scalar scores on a 1-5 Likert scale and (ii) criterion-specific natural-language rationales grounded in the rubric. Across 7 benchmarks and 5 task models, PEEM's accuracy axis strongly aligns with conventional accuracy while preserving model rankings (aggregate Spearman rho about 0.97, Pearson r about 0.94, p < 0.001). A multi-evaluator study with four models shows consistent relative judgments (pairwise rho = 0.68-0.85), supporting evaluator-agnostic deployment. Beyond alignment, PEEM captures complementary linguistic failure modes and remains informative under prompt perturbations: prompt-quality trends track downstream accuracy under iterative rewrites, semantic adversarial manipulations induce clear score degradation, and meaning-preserving paraphrases yield high stability (robustness rate about 76.7-80.6%). Finally, using only PEEM scores and rationales as feedback, a zero-shot prompt rewriting loop improves downstream accuracy by up to 11.7 points, outperforming supervised and RL-based prompt-optimization baselines. Overall, PEEM provides a reproducible, criterion-driven protocol that links prompt formulation to response behavior and enables systematic diagnosis and optimization of LLM interactions.

PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses

TL;DR

Overall, PEEM provides a reproducible, criterion-driven protocol that links prompt formulation to response behavior and enables systematic diagnosis and optimization of LLM interactions.

Abstract

Prompt design is a primary control interface for large language models (LLMs), yet standard evaluations largely reduce performance to answer correctness, obscuring why a prompt succeeds or fails and providing little actionable guidance. We propose PEEM (Prompt Engineering Evaluation Metrics), a unified framework for joint and interpretable evaluation of both prompts and responses. PEEM defines a structured rubric with 9 axes: 3 prompt criteria (clarity/structure, linguistic quality, fairness) and 6 response criteria (accuracy, coherence, relevance, objectivity, clarity, conciseness), and uses an LLM-based evaluator to output (i) scalar scores on a 1-5 Likert scale and (ii) criterion-specific natural-language rationales grounded in the rubric. Across 7 benchmarks and 5 task models, PEEM's accuracy axis strongly aligns with conventional accuracy while preserving model rankings (aggregate Spearman rho about 0.97, Pearson r about 0.94, p < 0.001). A multi-evaluator study with four models shows consistent relative judgments (pairwise rho = 0.68-0.85), supporting evaluator-agnostic deployment. Beyond alignment, PEEM captures complementary linguistic failure modes and remains informative under prompt perturbations: prompt-quality trends track downstream accuracy under iterative rewrites, semantic adversarial manipulations induce clear score degradation, and meaning-preserving paraphrases yield high stability (robustness rate about 76.7-80.6%). Finally, using only PEEM scores and rationales as feedback, a zero-shot prompt rewriting loop improves downstream accuracy by up to 11.7 points, outperforming supervised and RL-based prompt-optimization baselines. Overall, PEEM provides a reproducible, criterion-driven protocol that links prompt formulation to response behavior and enables systematic diagnosis and optimization of LLM interactions.
Paper Structure (52 sections, 4 equations, 2 figures, 30 tables, 2 algorithms)

This paper contains 52 sections, 4 equations, 2 figures, 30 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation of Prompt Engineering
  4. Interpretability in Prompt Engineering
  5. Method
  6. Preliminaries.
  7. PEEM Criteria Design
  8. Prompt Evaluation Criteria.
  9. Response Evaluation Criteria.
  10. PEEM Evaluation Procedure
  11. Experiments
  12. Experimental Setup
  13. Alignment with Existing Evaluation Metrics
  14. Correlation with Accuracy
  15. Cross-Evaluator Agreement
  16. ...and 37 more sections

Figures (2)

  • Figure 1: Comparison of traditional metrics and our proposed approach, PEEM. (a) Accuracy evaluates only whether the final answer is correct (e.g., "15"); (b) exact match (EM) requires the response to be identical to the reference answer. Both metrics ignore the reasoning process and linguistic quality. (c) PEEM evaluates the complete response across multiple axes---including logical consistency, contextual relevance, and clarity---while providing interpretable rationales for each score.
  • Figure 2: Overview of PEEM. The framework operates in two stages: (a) Prompt Evaluation assesses the input prompt $P = p \oplus q$ across three axes (Clarity, Linguistic Quality, Fairness), and (b) Response Evaluation assesses the model output $a$ across six axes (Accuracy, Coherence, Relevance, Objectivity, Clarity, Conciseness). For each axis, the LLM-based evaluator produces both a numerical score and a natural language rationale explaining the assessment.