Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimization

Yuanchen Wu, Saurabh Verma, Justin Lee, Fangzhou Xiong, Poppy Zhang, Amel Awadelkarim, Xu Chen, Yubai Yuan, Shawndra Hill

TL;DR

Prompt Duel Optimizer (PDO) reframes prompt optimization as a label-free dueling-bandit problem guided by LLM pairwise preferences. It combines Double Thompson Sampling to efficiently identify Copeland-optimal prompts with Top-Performer Guided Mutation to expand the search around strong candidates, achieving sample-efficient discovery without ground-truth labels. Empirical results on BBH and MS-MARCO show PDO consistently matching or surpassing label-free baselines and remaining competitive with supervised APO methods, with ablations confirming the value of both D-TS and mutation. The analysis also reveals how judge noise and partial labeling affect performance, and demonstrates the practicality of integrating a small amount of ground-truth labels when available.

Abstract

Large language models (LLMs) are highly sensitive to their input prompts, making prompt design a central challenge. While automatic prompt optimization (APO) reduces manual engineering, most approaches assume access to ground-truth references such as labeled validation data. In practice, however, collecting high-quality labels is costly and slow. We propose the Prompt Duel Optimizer (PDO), a sample-efficient framework for label-free prompt optimization. PDO formulates the problem as a dueling-bandit setting, where supervision signal comes from pairwise preference feedback provided by an LLM judge. The framework combines Double Thompson Sampling (D-TS), which prioritizes informative prompt comparisons, with Top-Performer Guided Mutation, which expands the candidate pool by mutating high-performing prompts. PDO naturally operates in label-free settings and can also incorporate partial labels to mitigate judge noise. Experiments on BIG-bench Hard (BBH) and MS MARCO show that PDO consistently outperforms baseline methods. Ablation studies further demonstrate the effectiveness of both D-TS and prompt mutation.

LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimization

TL;DR

Prompt Duel Optimizer (PDO) reframes prompt optimization as a label-free dueling-bandit problem guided by LLM pairwise preferences. It combines Double Thompson Sampling to efficiently identify Copeland-optimal prompts with Top-Performer Guided Mutation to expand the search around strong candidates, achieving sample-efficient discovery without ground-truth labels. Empirical results on BBH and MS-MARCO show PDO consistently matching or surpassing label-free baselines and remaining competitive with supervised APO methods, with ablations confirming the value of both D-TS and mutation. The analysis also reveals how judge noise and partial labeling affect performance, and demonstrates the practicality of integrating a small amount of ground-truth labels when available.

Abstract

Large language models (LLMs) are highly sensitive to their input prompts, making prompt design a central challenge. While automatic prompt optimization (APO) reduces manual engineering, most approaches assume access to ground-truth references such as labeled validation data. In practice, however, collecting high-quality labels is costly and slow. We propose the Prompt Duel Optimizer (PDO), a sample-efficient framework for label-free prompt optimization. PDO formulates the problem as a dueling-bandit setting, where supervision signal comes from pairwise preference feedback provided by an LLM judge. The framework combines Double Thompson Sampling (D-TS), which prioritizes informative prompt comparisons, with Top-Performer Guided Mutation, which expands the candidate pool by mutating high-performing prompts. PDO naturally operates in label-free settings and can also incorporate partial labels to mitigate judge noise. Experiments on BIG-bench Hard (BBH) and MS MARCO show that PDO consistently outperforms baseline methods. Ablation studies further demonstrate the effectiveness of both D-TS and prompt mutation.
Paper Structure (51 sections, 1 theorem, 13 equations, 8 figures, 4 tables, 1 algorithm)

This paper contains 51 sections, 1 theorem, 13 equations, 8 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and Problem Setup
  3. Background: Dueling Bandits
  4. Prompt Optimization Through the Lens of Dueling Bandits
  5. Problem Setup.
  6. Prompt Optimization Objective.
  7. Prompt Duel Optimizer (PDO)
  8. Efficient Prompt Selection via Double Thompson Sampling
  9. Notations.
  10. Per-round Procedure ($t=1,2,\dots,T$):
  11. Implications.
  12. Efficient Prompt Discovery via Top-Performer Guided Mutation
  13. Mutation Procedure.
  14. Coverage-Theoretic Justification.
  15. Experiment and Results
  16. ...and 36 more sections

Key Result

Theorem 1

Let $\mathcal{P}=\{p_1,\dots,p_K\}$ be $K$ prompts with pairwise preference probabilities $\mu_{i,j}=\Pr(p_i\succ p_j)$. Define the normalized Copeland score If at round $t$ the algorithm selects $(a_t,b_t)$, the regret is $r_t=\zeta^*-\max\{\zeta_{a_t},\zeta_{b_t}\}$ and $R_T=\sum_{t=1}^T r_t$. Then Double Thompson Sampling (D-TS) achieves wu2016double

Figures (8)

  • Figure 1: Workflow of the Prompt Duel Optimizer.
  • Figure 2: Test performance of the winning prompt on the four MS-MARCO tasks. Each curve shows the mean score of the current Copeland leader over rounds, with 50 duels per round. PDO with D-TS consistently outperforms RUCB and Random sampling, and surpasses the SPO baseline within a few rounds. Gray lines indicate the median test score across all prompts generated by PDO.
  • Figure 3: The accuracy of the LLM judge varies across tasks, with some tasks (e.g., Geometric) showing persistent misjudgments compared to an oracle judge. Introducing a fraction of true labels ($r\in\{0.3,0.5\}$) helps reduce the impact of noisy judgments and guide D-TS more quickly toward the top-ranked prompts.
  • Figure 4: On the Numeric task, Llama-70B as judge enables D-TS to find strong prompts faster and reach higher scores, whereas on the Description task, Llama-8B slightly outperforms 70B overall.
  • Figure 5: Comparison of pairwise preference used in PDO and pointwise scoring for prompt selection on MS-MARCO. Results indicate that pairwise preference leads to higher-performing prompts across model sizes (7 out of 8 cases).
  • ...and 3 more figures

Theorems & Definitions (1)

  • Theorem 1: PDO Copeland Regret