PromptWizard: Task-Aware Prompt Optimization Framework
Eshaan Agarwal, Joykirat Singh, Vivek Dani, Raghav Magazine, Tanuja Ganu, Akshay Nambi
TL;DR
PromptWizard automates discrete prompt optimization via a self-evolving feedback loop that mutates, critiques, and synthesizes both instructions and in-context examples. By jointly refining prompts and demonstrations and leveraging chain-of-thought reasoning, PW achieves superior performance across 45 tasks, including BBII and BBH, while maintaining efficiency. The framework demonstrates strong zero-shot and few-shot gains and shows robustness across base LLMs and even smaller models, with substantial cost reductions in API calls and tokens. Collectively, PW advances prompt engineering toward scalable, interpretable, and cost-effective automation suitable for diverse domains.
Abstract
Large language models (LLMs) have transformed AI across diverse domains, with prompting being central to their success in guiding model outputs. However, manual prompt engineering is both labor-intensive and domain-specific, necessitating the need for automated solutions. We introduce PromptWizard, a novel, fully automated framework for discrete prompt optimization, utilizing a self-evolving, self-adapting mechanism. Through a feedback-driven critique and synthesis process, PromptWizard achieves an effective balance between exploration and exploitation, iteratively refining both prompt instructions and in-context examples to generate human-readable, task-specific prompts. This guided approach systematically improves prompt quality, resulting in superior performance across 45 tasks. PromptWizard excels even with limited training data, smaller LLMs, and various LLM architectures. Additionally, our cost analysis reveals a substantial reduction in API calls, token usage, and overall cost, demonstrating PromptWizard's efficiency, scalability, and advantages over existing prompt optimization strategies.