Prompt Recursive Search: A Living Framework with Adaptive Growth in LLM Auto-Prompting

TL;DR

This work addresses inefficiencies and error accumulation in prompting large language models by formalizing a complexity-aware framework called Prompt Recursive Search (PRS). PRS combines the strengths of Expert-Designed Prompts and LLM-Derived Prompts, using a numerical complexity score to guide recursive planning and solution decomposition, thereby reducing token waste and controlling error propagation. Empirical results on the BBH benchmark show PRS improves accuracy over CoT (e.g., on the Llama3-7B model, $8\%$ absolute and $22\%$ relative in one comparison), and ablations on Yi-34B demonstrate consistent gains, indicating robustness across model scales. The framework offers a scalable, adaptive approach to prompt design with practical impact for multi-domain reasoning tasks in structured evaluation settings, while acknowledging limitations such as format enforcement and variability in LLM outputs. $\text{Steps} \approx \frac{\text{Complexity}}{2}$ plays a central role in guiding the planning process, enabling PRS to adaptively grow with problem difficulty.$

Abstract

Large Language Models (LLMs) exhibit remarkable proficiency in addressing a diverse array of tasks within the Natural Language Processing (NLP) domain, with various prompt design strategies significantly augmenting their capabilities. However, these prompts, while beneficial, each possess inherent limitations. The primary prompt design methodologies are twofold: The first, exemplified by the Chain of Thought (CoT), involves manually crafting prompts specific to individual datasets, hence termed Expert-Designed Prompts (EDPs). Once these prompts are established, they are unalterable, and their effectiveness is capped by the expertise of the human designers. When applied to LLMs, the static nature of EDPs results in a uniform approach to both simple and complex problems within the same dataset, leading to the inefficient use of tokens for straightforward issues. The second method involves prompts autonomously generated by the LLM, known as LLM-Derived Prompts (LDPs), which provide tailored solutions to specific problems, mitigating the limitations of EDPs. However, LDPs may encounter a decline in performance when tackling complex problems due to the potential for error accumulation during the solution planning process. To address these challenges, we have conceived a novel Prompt Recursive Search (PRS) framework that leverages the LLM to generate solutions specific to the problem, thereby conserving tokens. The framework incorporates an assessment of problem complexity and an adjustable structure, ensuring a reduction in the likelihood of errors. We have substantiated the efficacy of PRS framework through extensive experiments using LLMs with different numbers of parameters across a spectrum of datasets in various domains. Compared to the CoT method, the PRS method has increased the accuracy on the BBH dataset by 8% using Llama3-7B model, achieving a 22% improvement.

Figures (3)

• Figure 1: Pipeline of our proposed method Prompt Recursive Search with complexity score limitation (PRS).
• Figure 2: Comparison between PRS (ours) and 0-shot-cot on the BBH dataset.
• Figure 3: The ablation experiments based on the Yi-34B model on the BBH dataset. The results represent metrics on the formal_fallacies subset.