Submodular Evaluation Subset Selection in Automatic Prompt Optimization

TL;DR

This work studies evaluation subset selection for prompt optimization from a principled perspective and proposes SESS, a submodular evaluation subset selection method that is monotone and submodular, enabling greedy selection with theoretical guarantees.

Abstract

Automatic prompt optimization reduces manual prompt engineering, but relies on task performance measured on a small, often randomly sampled evaluation subset as its main source of feedback signal. Despite this, how to select that evaluation subset is usually treated as an implementation detail. We study evaluation subset selection for prompt optimization from a principled perspective and propose SESS, a submodular evaluation subset selection method. We frame selection as maximizing an objective set function and show that, under mild conditions, it is monotone and submodular, enabling greedy selection with theoretical guarantees. Across GSM8K, MATH, and GPQA-Diamond, submodularly selected evaluation subsets can yield better optimized prompts than random or heuristic baselines.

Figures (4)

• Figure 1: The general framework of APO. SESS replaces random or heuristic subset selection with a principled submodular evaluation subset selection module.
• Figure 2: Verbal confidence elicitation prompt
• Figure 3: Multiple-choice answer prompt used for likelihood-based confidence.
• Figure 4: Numeric/free-form answer prompt used for likelihood-based confidence.

Theorems & Definitions (2)

• proof : Proof for submodularity of $\mathcal{F}_{\mathrm{rep}}$ and $\mathcal{F}_{\mathrm{wrep}}$
• proof : Proof for monotonicity of $\mathcal{F}_{\mathrm{rep}}$ and $\mathcal{F}_{\mathrm{wrep}}$