Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimization

TL;DR

CFPO introduces a joint framework for Content-Format Integrated Prompt Optimization, addressing the notable impact of prompt formatting on LLM performance. It deploys a structured prompt template that decouples content from format, a component-wise content optimizer enriched by correct-case analysis, and a dynamic format optimizer that builds a hierarchical format pool (with scoring) and uses LLM-assisted generation plus a UCT search to discover effective formats. Across multiple open-source LLMs and tasks, CFPO demonstrates measurable gains over content-only baselines, with ablations confirming the synergy between content and format optimization, especially in reasoning tasks. The approach is model-agnostic and scalable, offering a practical path to more robust prompting with publicly available code.

Abstract

Large Language Models (LLMs) have shown significant capability across various tasks, with their real-world effectiveness often driven by prompt design. While recent research has focused on optimizing prompt content, the role of prompt formatting, a critical but often overlooked dimension, has received limited systematic investigation. In this paper, we introduce Content-Format Integrated Prompt Optimization (CFPO), an innovative methodology that jointly optimizes both prompt content and formatting through an iterative refinement process. CFPO leverages natural language mutations to explore content variations and employs a dynamic format exploration strategy that systematically evaluates diverse format options. Our extensive evaluations across multiple tasks and open-source LLMs demonstrate that CFPO demonstrates measurable performance improvements compared to content-only optimization methods. This highlights the importance of integrated content-format optimization and offers a practical, model-agnostic approach to enhancing LLM performance. Code is available at https://github.com/HenryLau7/CFPO.

Figures (6)

• Figure 1: The crucial role of prompt formatting and its interaction with content. (A): Model-specific format sensitivity: LLM performance varies significantly across different prompt formats on GSM8K task. (B): Content-format interdependence: The optimal format for a prompt depends on its content, highlighting the need for joint optimization.
• Figure 2: Illustration of the CFPO pipeline within a single iteration round. In the initial Component-wise Content Optimization stage, case-diagnosis and Monte-Carlo sampling are employed for content mutation. Subsequently, the Format Optimization stage identifies the most suitable format for each content candidate. The yellow dashed line indicates where the LLM optimizer is employed to guide the optimization process.
• Figure 3: An illustrative example of our Structured Prompt Template. This template systematically organizes the prompt into distinct components, each serving a specific functional role. When formulating a prompt, the template first employs a Query format to present examples and queries, and then integrates all content components via the Prompt Renderer to construct the comprehensive prompt string.
• Figure 4: Built-in formats and rendering effects in our initial format pool. The final format configuration is achieved by selecting and combining elements from both the Prompt Renderer and the Query Format categories.
• Figure 5: Overview of in-context examples and text lengths for optimized prompts on various tasks and models.