Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UPA: Unsupervised Prompt Agent via Tree-Based Search and Selection

Siran Peng, Weisong Zhao, Tianyu Fu, Chenxu Zhao, Tianshuo Zhang, Haoyuan Zhang, Xiangyu Zhu, Minghui Wu, Zhen Lei

TL;DR

UPA tackles unsupervised prompt optimization by combining a tree-based search with relative LLM judgments and a two-stage BTL-based selection. It decouples exploration from final ranking, using path-wise Bayesian filtering to prune candidates and a global BT maximization to select the best prompt. Empirical results across diverse benchmarks show UPA outperforms state-of-the-art prompt optimization approaches and generalizes to multiple frontier LLMs, highlighting the effectiveness of unsupervised, agent-style prompt refinement. The approach provides a principled framework for robust prompt discovery when explicit reward signals are unavailable, with practical implications for scalable LLM tuning and deployment.

Abstract

Prompt agents have recently emerged as a promising paradigm for automated prompt optimization, framing refinement as a sequential decision-making problem over a structured prompt space. While this formulation enables the use of advanced planning algorithms, these methods typically assume access to supervised reward signals, which are often unavailable in practical scenarios. In this work, we propose UPA, an Unsupervised Prompt Agent that realizes structured search and selection without relying on supervised feedback. Specifically, during search, UPA iteratively constructs an evolving tree structure to navigate the prompt space, guided by fine-grained and order-invariant pairwise comparisons from Large Language Models (LLMs). Crucially, as these local comparisons do not inherently yield a consistent global scale, we decouple systematic prompt exploration from final selection, introducing a two-stage framework grounded in the Bradley-Terry-Luce (BTL) model. This framework first performs path-wise Bayesian aggregation of local comparisons to filter candidates under uncertainty, followed by global tournament-style comparisons to infer latent prompt quality and identify the optimal prompt. Experiments across multiple tasks demonstrate that UPA consistently outperforms existing prompt optimization methods, showing that agent-style optimization remains highly effective even in fully unsupervised settings.

UPA: Unsupervised Prompt Agent via Tree-Based Search and Selection

TL;DR

UPA tackles unsupervised prompt optimization by combining a tree-based search with relative LLM judgments and a two-stage BTL-based selection. It decouples exploration from final ranking, using path-wise Bayesian filtering to prune candidates and a global BT maximization to select the best prompt. Empirical results across diverse benchmarks show UPA outperforms state-of-the-art prompt optimization approaches and generalizes to multiple frontier LLMs, highlighting the effectiveness of unsupervised, agent-style prompt refinement. The approach provides a principled framework for robust prompt discovery when explicit reward signals are unavailable, with practical implications for scalable LLM tuning and deployment.

Abstract

Prompt agents have recently emerged as a promising paradigm for automated prompt optimization, framing refinement as a sequential decision-making problem over a structured prompt space. While this formulation enables the use of advanced planning algorithms, these methods typically assume access to supervised reward signals, which are often unavailable in practical scenarios. In this work, we propose UPA, an Unsupervised Prompt Agent that realizes structured search and selection without relying on supervised feedback. Specifically, during search, UPA iteratively constructs an evolving tree structure to navigate the prompt space, guided by fine-grained and order-invariant pairwise comparisons from Large Language Models (LLMs). Crucially, as these local comparisons do not inherently yield a consistent global scale, we decouple systematic prompt exploration from final selection, introducing a two-stage framework grounded in the Bradley-Terry-Luce (BTL) model. This framework first performs path-wise Bayesian aggregation of local comparisons to filter candidates under uncertainty, followed by global tournament-style comparisons to infer latent prompt quality and identify the optimal prompt. Experiments across multiple tasks demonstrate that UPA consistently outperforms existing prompt optimization methods, showing that agent-style optimization remains highly effective even in fully unsupervised settings.
Paper Structure (27 sections, 1 theorem, 18 equations, 4 figures, 9 tables, 2 algorithms)

This paper contains 27 sections, 1 theorem, 18 equations, 4 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Setting & Overview
  4. Pairwise Comparison with Bias Mitigation
  5. Tree-Based Search with Relative Feedback
  6. Latent Quality Modeling via BTL
  7. Stage I Selection: Path-Wise Bayesian Filtering
  8. Stage II Selection: Global BTL Maximization
  9. Experiments
  10. Setup
  11. Performance
  12. Ablation Studies
  13. Conclusion
  14. Related Works
  15. Prompt Engineering
  16. ...and 12 more sections

Key Result

Proposition 2.2

Under the BTL model, the path-wise posterior mean $\mu_v$ provides a consistent estimate of the true latent quality difference $\theta_v - \theta_o$. Moreover, under Assumption ass:ind, as the sampling budget on each edge increases, the path-wise variance $\sigma_v^2$ vanishes. Consequently, the unc

Figures (4)

  • Figure 1: Comparison of prompt optimization paradigms, including most Prompt Optimization (PO) approaches, existing prompt agents wangpromptagentyu2025optimizing, SPO xiang2025self, and our UPA. Unlike prior methods, UPA uniquely achieves structured exploration in a fully ground-truth (GT) free setting.
  • Figure 2: Overview of UPA. Our method decouples prompt optimization into two phases: search and selection. During the search phase, UPA explores the prompt space using a tree-based framework driven by relative feedback. In the selection phase, it employs a two-stage strategy: path-wise Bayesian filtering to prune the search tree, followed by global BTL maximization to infer the optimal prompt.
  • Figure 3: Sensitivity analysis of hyperparameters. The top and bottom rows show strategy-oriented and scale-oriented parameters.
  • Figure 4: Visualization of completed search trees across five closed-ended tasks. Nodes are indexed by their expansion iteration.

Theorems & Definitions (2)

  • Proposition 2.2
  • proof