Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL

TL;DR

The paper tackles the challenge that optimal prompts for arithmetic reasoning are query-dependent, proposing a query-aware, offline approach. It introduces Prompt-OIRL, which leverages offline inverse reinforcement learning to learn a reward model from demonstrations of existing prompts, then uses a best-of-N strategy to select the most effective prompt at inference time without querying the LLM. Across GSM8K, MAWPS, and SVAMP datasets and three diverse LLMs, Prompt-OIRL demonstrates strong gains in accuracy and robustness, while dramatically reducing inference costs by avoiding repeated LLM calls. The work highlights the value of offline prompting data, provides detailed cost analyses, and suggests directions for broader offline prompt evaluation and optimization beyond arithmetic reasoning.

Abstract

In this study, we aim to enhance the arithmetic reasoning ability of Large Language Models (LLMs) through zero-shot prompt optimization. We identify a previously overlooked objective of query dependency in such optimization and elucidate two ensuing challenges that impede the successful and economical design of prompt optimization techniques. One primary issue is the absence of an effective method to evaluate prompts during inference when the golden answer is unavailable. Concurrently, learning via interactions with the LLMs to navigate the expansive natural language prompting space proves to be resource-intensive. To address this, we introduce Prompt-OIRL, which harnesses offline inverse reinforcement learning to draw insights from offline prompting demonstration data. Such data exists as by-products when diverse prompts are benchmarked on open-accessible datasets. With Prompt-OIRL, the query-dependent prompt optimization objective is achieved by first learning an offline reward model. This model can evaluate any query-prompt pairs without accessing LLMs. Subsequently, a best-of-N strategy is deployed to recommend the optimal prompt. Our experimental evaluations across various LLM scales and arithmetic reasoning datasets underscore both the efficacy and economic viability of the proposed approach.

Figures (26)

• Figure 1: A motivating example.(https://chat.openai.com/share/0f2d11b1-322a-4c47-a877-ad6fbace8179, https://chat.openai.com/share/15870a47-93c7-4b98-96c8-af0516c0c999) No prompt is perfect that works for all queries. The optimal prompt is query-dependent. Yet the seeking of such prompts is hindered by the Challenges 1-2 we identified. Our method optimizes prompt during inference on a query-dependent level effectively and cost-efficiently.
• Figure 2: The Adjusted Objective and Challenges in prompt optimization. We use blue to denote fixed functions, pink for datasets, and green for functions to be optimized. Solid lines show the flow of outputs, and dashed lines denote the learning process.
• Figure 3: The offline demonstration dataset is generated as a by-product of evaluating existing (query-agnostic) prompts.
• Figure 4: Prompt-OIRL addresses the specified Objective and challenges. It first learns a proxy reward model from the offline demonstration dataset we created in the last section. Such a learned reward model can be applied in inference time to evaluate prompts in a query-dependent manner without access to the language model, hence optimizing prompt w.r.t. such a proxy reward model solves all issues identified.
• Figure 5: Performance of Prompt-OIRL under two typical settings. On both settings with scarce (i.e., 1 training prompt) and rich demonstration data (i.e., 6 training prompts), Prompt-OIRL achieves better performance.