Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When "Better" Prompts Hurt: Evaluation-Driven Iteration for LLM Applications

Daniel Commey

TL;DR

The paper confronts the challenge of evaluating LLM-based applications, which are non-deterministic and high-dimensional, by proposing an evaluation-driven loop (Define-Test-Diagnose-Fix) paired with the Minimum Viable Evaluation Suite (MVES) across general, retrieval-based, and agentic use cases. It synthesizes evaluation methods (automated offline checks, human rubrics, and LLM-as-judge), introduces a quality taxonomy (correctness, helpfulness, harmlessness, groundedness, etc.), and analyzes LLM-as-judge failure modes to calibrate judgments. A key empirical finding is that task-specific prompts can outperform generic improvements on structured tasks, underscoring the need for regression-tested, task-aligned prompt iteration and explicit claim calibration rather than universal prompt recipes. The work provides reproducible artifacts, case studies, and best-practice guidance to standardize application-level evaluation and guide production monitoring, with directions toward standardized benchmarks, improved judges, and automated red-teaming. Overall, the framework enables reliable deployment of LLM applications through systematic evaluation-driven iteration and robust testing infrastructure.

Abstract

Evaluating Large Language Model (LLM) applications differs from traditional software testing because outputs are stochastic, high-dimensional, and sensitive to prompt and model changes. We present an evaluation-driven workflow - Define, Test, Diagnose, Fix - that turns these challenges into a repeatable engineering loop. We introduce the Minimum Viable Evaluation Suite (MVES), a tiered set of recommended evaluation components for (i) general LLM applications, (ii) retrieval-augmented generation (RAG), and (iii) agentic tool-use workflows. We also synthesize common evaluation methods (automated checks, human rubrics, and LLM-as-judge) and discuss known judge failure modes. In reproducible local experiments (Ollama; Llama 3 8B Instruct and Qwen 2.5 7B Instruct), we observe that a generic "improved" prompt template can trade off behaviors: on our small structured suites, extraction pass rate decreased from 100% to 90% and RAG compliance from 93.3% to 80% for Llama 3 when replacing task-specific prompts with generic rules, while instruction-following improved. These findings motivate evaluation-driven prompt iteration and careful claim calibration rather than universal prompt recipes. All test suites, harnesses, and results are included for reproducibility.

When "Better" Prompts Hurt: Evaluation-Driven Iteration for LLM Applications

TL;DR

The paper confronts the challenge of evaluating LLM-based applications, which are non-deterministic and high-dimensional, by proposing an evaluation-driven loop (Define-Test-Diagnose-Fix) paired with the Minimum Viable Evaluation Suite (MVES) across general, retrieval-based, and agentic use cases. It synthesizes evaluation methods (automated offline checks, human rubrics, and LLM-as-judge), introduces a quality taxonomy (correctness, helpfulness, harmlessness, groundedness, etc.), and analyzes LLM-as-judge failure modes to calibrate judgments. A key empirical finding is that task-specific prompts can outperform generic improvements on structured tasks, underscoring the need for regression-tested, task-aligned prompt iteration and explicit claim calibration rather than universal prompt recipes. The work provides reproducible artifacts, case studies, and best-practice guidance to standardize application-level evaluation and guide production monitoring, with directions toward standardized benchmarks, improved judges, and automated red-teaming. Overall, the framework enables reliable deployment of LLM applications through systematic evaluation-driven iteration and robust testing infrastructure.

Abstract

Evaluating Large Language Model (LLM) applications differs from traditional software testing because outputs are stochastic, high-dimensional, and sensitive to prompt and model changes. We present an evaluation-driven workflow - Define, Test, Diagnose, Fix - that turns these challenges into a repeatable engineering loop. We introduce the Minimum Viable Evaluation Suite (MVES), a tiered set of recommended evaluation components for (i) general LLM applications, (ii) retrieval-augmented generation (RAG), and (iii) agentic tool-use workflows. We also synthesize common evaluation methods (automated checks, human rubrics, and LLM-as-judge) and discuss known judge failure modes. In reproducible local experiments (Ollama; Llama 3 8B Instruct and Qwen 2.5 7B Instruct), we observe that a generic "improved" prompt template can trade off behaviors: on our small structured suites, extraction pass rate decreased from 100% to 90% and RAG compliance from 93.3% to 80% for Llama 3 when replacing task-specific prompts with generic rules, while instruction-following improved. These findings motivate evaluation-driven prompt iteration and careful claim calibration rather than universal prompt recipes. All test suites, harnesses, and results are included for reproducibility.
Paper Structure (133 sections, 1 equation, 1 figure, 13 tables)

This paper contains 133 sections, 1 equation, 1 figure, 13 tables.

Table of Contents

  1. Keywords.
  2. Contributions.
  3. Introduction
  4. Why LLM Evaluation Differs from Traditional Testing
  5. Key Risks in LLM Applications
  6. What This Paper Provides
  7. Artifacts.
  8. The Evaluation Loop
  9. The Core Workflow
  10. Translating Requirements into Tests
  11. The Role of Golden Sets
  12. Iteration in Practice
  13. Connecting Offline and Online Evaluation
  14. Quality Taxonomy
  15. Correctness
  16. ...and 118 more sections

Figures (1)

  • Figure 1: Evaluation pipeline overview: inputs flow through the application to produce model outputs, which are evaluated (automated checks, LLM-as-judge, and/or human evaluation) and aggregated into monitoring signals that drive iteration.