FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema

TL;DR

FIPO introduces a fully offline, local optimizer for free-form instruction-oriented prompt optimization, addressing privacy and generalization gaps in online APO. It relies on a modular APO template and a large Prompt Optimization Preference (POP) dataset to train a general local optimizer (M_o) that can enhance prompts for any testing generator without exposing data to external LLMs. Through diverse fine-tuning strategies (SFT, DPO, IPO, IPL) and dataset diversification, FIPO achieves general performance gains across five public benchmarks and multiple models, often outperforming existing ad-hoc APO baselines. The work demonstrates practical impact by enabling model-agnostic, cost-efficient prompt optimization and provides insights into data design, training strategies, and case-level improvements, while acknowledging limitations like overly explicit guidance and evaluation metrics beyond accuracy.

Abstract

When the quality of naive prompts is carefully optimized by human experts, the task performance of large language models (LLMs) can be significantly improved. However, expert-based prompt optimizations are expensive. Herein, some works have proposed Automatic Prompt Optimization (APO), to optimize naive prompts according to task outputs of given in-box testing models, with the help of advanced LLMs (e.g., GPT-4) in an ad-hoc way. Although effective, existing schemes suffer from poor generalization ability and privacy risk. To this end, we collect the first large-scale Prompt Optimization Preference dataset (POP), fine-tune offline local LLM-based optimizers, then fairly test with various downstream models. Our method allows accurate optimization of the core task instruction part within the naive prompt in a model-agnostic manner, and thus is named Free-from Instruction-oriented Prompt Optimization (FIPO). In specific, FIPO uses a modular APO template that dynamically integrate the naive task instruction, optional instruction responses, and optional ground truth to produce finely optimized prompts. The POP dataset is meticulously constructed using advanced LLMs, undergoing rigorous cross-validation by human experts and analytical models. Leveraging insights from the data with Tulu2 models and diverse fine-tuning strategies, we validate the efficacy of FIPO framework across five public benchmarks and six testing models. Check codes and data here: https://github.com/LuJunru/FIPO_Project.

Figures (6)

• Figure 1: Online Ad-hoc APO vs. our Local End-to-End FIPO: Although both approaches leverage advanced LLMs (e.g., GPT-4), FIPO introduces a a locally trained pipeline that eliminates any dependence on in-box model generators, ensuring a fully self-contained and end-to-end optimization process.
• Figure 2: Step 1 and 2 of FIPO: (1) Design a meta-template for universal APO; (2) Collect 30,000 large-scale prompt optimization preference exemples using a suboptimal LLM (GPT-3.5-turbo) and an optimal LLM (GPT-4).
• Figure 3: Step 3 of FIPO: transitional dataset diversification and several mainstream fine-tuning strategies.
• Figure 4: The FIPO-optimized prompts help various downstream testing LLMs (X-axis) gain more promising improvements, compared with other prompt optimization approaches (shown by the bars). We specifically annotate the improvements of FIPO against naive prompts from the original dataset (↑). More details: Appendix \ref{['app:cost']} and \ref{['app:bbh']}.
• Figure 5: An overview of our dataset diversification step. It is recommended to view the details with colors.