PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMs

TL;DR

PRESTO reframes instruction optimization for black-box LLMs by exploiting the many-to-one mapping from soft prompts to produced instructions. It introduces a preimage-informed framework with three components: score sharing across soft-prompts that map to the same instruction, preimage-based initialization to maximize coverage of the search space, and score consistency regularization to stabilize the score predictor within preimages. Empirically, PRESTO achieves state-of-the-art performance on 30 instruction-induction tasks and 3 arithmetic tasks, and demonstrates strong transfer to chain-of-thought prompting. The approach combines data-efficient query reuse with principled initialization and regularization, yielding robust gains across diverse tasks and model pairings, with practical computational efficiency.

Abstract

Large language models (LLMs) have achieved remarkable success across diverse domains, due to their strong instruction-following capabilities. This has led to increasing interest in optimizing instructions for black-box LLMs, whose internal parameters are inaccessible but widely used due to their strong performance. To optimize instructions for black-box LLMs, recent methods employ white-box LLMs to generate candidate instructions from optimized soft prompts. However, white-box LLMs often map different soft prompts to the same instruction, leading to redundant queries. While previous studies regarded this many-to-one mapping as a structure that hinders optimization efficiency, we reinterpret it as a useful prior knowledge that can accelerate the optimization. To this end, we introduce PREimage-informed inSTruction Optimization (PRESTO), a novel framework that leverages the preimage structure of soft prompts for efficient optimization. PRESTO consists of three key components: (1) score sharing, which shares the evaluation score with all soft prompts in a preimage; (2) preimage-based initialization, which selects initial data points that maximize search space coverage using preimage information; and (3) score consistency regularization, which enforces prediction consistency within each preimage. By leveraging preimages, PRESTO achieves the effect of effectively obtaining 14 times more scored data under the same query budget, resulting in more efficient optimization. Experimental results on 33 instruction optimization tasks demonstrate the superior performance of PRESTO. Code is available at https://github.com/mlvlab/PRESTO

Figures (10)

• Figure 1: Motivating observations illustrating the many-to-one mapping from soft prompts to instructions in a white-box LLM (LLaMA3.1-8B-Instruct grattafiori2024llama). Figure \ref{['subfig:unique_instruction']} shows that the white-box LLM produces approximately 6,500 unique instructions from 10,000 distinct soft prompts. Figure \ref{['subfig:group_size']} presents the distribution of preimage sizes, displaying the top 100 largest preimages. The largest preimage contains more than 1,000 soft prompts, while the 100th largest has around 5. Both figures report the average experimental results over the instruction induction tasks used in Table \ref{['tab:main results']}.
• Figure 2: The overall process of our proposed PRESTO framework. It consists of two main stages: initialization and optimization. In the initialization stage, our method performs ① preimage-based score sharing (Section \ref{['main_sec:ScoreAssignment']}) and ② preimage-based initialization to improve search space coverage (Section \ref{['main_sec:GroupInitialization']}). For the optimization stage, we train the score predictor with ③ score consistency regularization (Section \ref{['main_sec:NeuralUCB']}) and we apply ① preimage-based score sharing to share scores of newly observed data within the same preimage.
• Figure 3: Toy example comparing models trained w/o and w/ our consistency loss $\mathcal{L}_{\text{cons}}$ in Eq. \ref{['eq:consistency_loss']}.
• Figure 4: Average number of scored soft prompts after optimization across all tasks.
• Figure 5: Performance of score predictor trained with diverse methods.