Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Error Taxonomy-Guided Prompt Optimization

Mayank Singh, Vikas Yadav, Eduardo Blanco

TL;DR

Error Taxonomy-Guided Prompt Optimization (ETGPO) introduces a top-down APO framework that builds a global error taxonomy from backbone LLM traces and generates targeted guidance for prevalent failure modes. By focusing on the most frequent errors and augmenting prompts with actionable guidance, ETGPO achieves competitive accuracy across mathematics, QA, and logical reasoning tasks while using roughly one-third the optimization budget. Experiments on six datasets show ETGPO matching or surpassing state-of-the-art APO methods, with notable gains on FOLIO and substantial efficiency improvements. The approach also provides an interpretable view of the target model's weaknesses, reducing overfitting risk by emphasizing global error patterns.

Abstract

Automatic Prompt Optimization (APO) is a powerful approach for extracting performance from large language models without modifying their weights. Many existing methods rely on trial-and-error, testing different prompts or in-context examples until a good configuration emerges, often consuming substantial compute. Recently, natural language feedback derived from execution logs has shown promise as a way to identify how prompts can be improved. However, most prior approaches operate in a bottom-up manner, iteratively adjusting the prompt based on feedback from individual problems, which can cause them to lose the global perspective. In this work, we propose Error Taxonomy-Guided Prompt Optimization (ETGPO), a prompt optimization algorithm that adopts a top-down approach. ETGPO focuses on the global failure landscape by collecting model errors, categorizing them into a taxonomy, and augmenting the prompt with guidance targeting the most frequent failure modes. Across multiple benchmarks spanning mathematics, question answering, and logical reasoning, ETGPO achieves accuracy that is comparable to or better than state-of-the-art methods, while requiring roughly one third of the optimization-phase token usage and evaluation budget.

Error Taxonomy-Guided Prompt Optimization

TL;DR

Error Taxonomy-Guided Prompt Optimization (ETGPO) introduces a top-down APO framework that builds a global error taxonomy from backbone LLM traces and generates targeted guidance for prevalent failure modes. By focusing on the most frequent errors and augmenting prompts with actionable guidance, ETGPO achieves competitive accuracy across mathematics, QA, and logical reasoning tasks while using roughly one-third the optimization budget. Experiments on six datasets show ETGPO matching or surpassing state-of-the-art APO methods, with notable gains on FOLIO and substantial efficiency improvements. The approach also provides an interpretable view of the target model's weaknesses, reducing overfitting risk by emphasizing global error patterns.

Abstract

Automatic Prompt Optimization (APO) is a powerful approach for extracting performance from large language models without modifying their weights. Many existing methods rely on trial-and-error, testing different prompts or in-context examples until a good configuration emerges, often consuming substantial compute. Recently, natural language feedback derived from execution logs has shown promise as a way to identify how prompts can be improved. However, most prior approaches operate in a bottom-up manner, iteratively adjusting the prompt based on feedback from individual problems, which can cause them to lose the global perspective. In this work, we propose Error Taxonomy-Guided Prompt Optimization (ETGPO), a prompt optimization algorithm that adopts a top-down approach. ETGPO focuses on the global failure landscape by collecting model errors, categorizing them into a taxonomy, and augmenting the prompt with guidance targeting the most frequent failure modes. Across multiple benchmarks spanning mathematics, question answering, and logical reasoning, ETGPO achieves accuracy that is comparable to or better than state-of-the-art methods, while requiring roughly one third of the optimization-phase token usage and evaluation budget.
Paper Structure (25 sections, 14 figures, 7 tables, 1 algorithm)

This paper contains 25 sections, 14 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Error Collection
  5. Error Taxonomy Creation
  6. Error Category Selection
  7. Guidance Generation
  8. Evaluation
  9. Datasets
  10. Baselines
  11. Experimental Results
  12. Accuracy results
  13. Token usage results
  14. Detailed Analysis
  15. Number of error collection runs
  16. ...and 10 more sections

Figures (14)

  • Figure 1: Overview of ETGPO. We first collect errors from failed traces and then organize them into a taxonomy. After filtering to the most prevalent error categories, we generate actionable guidance for each and combine them into the final prompt. This entire process is automated end-to-end.
  • Figure 2: Comparison between the original system prompt and the ETGPO-optimized prompt for AIME, using GPT-4.1-mini as the backbone model and GPT-4.1 as the optimizer model. The complete optimized prompt has 10 blocks, of which only the first block and a portion of the second are shown.
  • Figure 3: Count of failed traces before and after optimization with ETGPO for the top 6 error categories from the error taxonomy on the validation sets of (a) AIME and (b) FOLIO. All but one category shows fewer failed traces, supporting our hypothesis that targeted guidance reduces error prevalence.
  • Figure 4: Error taxonomy creation prompt for first batch (Part 1 of 2)
  • Figure 5: Error taxonomy creation prompt for first batch (Part 2 of 2)
  • ...and 9 more figures