SPRIG: Improving Large Language Model Performance by System Prompt Optimization

TL;DR

This work introduces SPRIG, an edit-based genetic algorithm for optimizing system prompts to achieve broad generalization across tasks, languages, and model families. By constructing a large component corpus and iteratively editing prompts with UCB-guided selection, Sprig yields performance gains comparable to task-specific optimization and complements it when combined. The results show robust cross-model and multilingual generalization, though gains are less pronounced when transferring to larger model sizes. The study provides practical insights into the global role of system-level instructions in maximizing LLM potential, while acknowledging computational costs and ethical considerations. Overall, Sprig offers a promising, single-system-prompt approach to boost LLM performance across diverse domains.

Abstract

Large Language Models (LLMs) have shown impressive capabilities in many scenarios, but their performance depends, in part, on the choice of prompt. Past research has focused on optimizing prompts specific to a task. However, much less attention has been given to optimizing the general instructions included in a prompt, known as a system prompt. To address this gap, we propose SPRIG, an edit-based genetic algorithm that iteratively constructs prompts from prespecified components to maximize the model's performance in general scenarios. We evaluate the performance of system prompts on a collection of 47 different types of tasks to ensure generalizability. Our study finds that a single optimized system prompt performs on par with task prompts optimized for each individual task. Moreover, combining system and task-level optimizations leads to further improvement, which showcases their complementary nature. Experiments also reveal that the optimized system prompts generalize effectively across model families, parameter sizes, and languages. This study provides insights into the role of system-level instructions in maximizing LLM potential.

Figures (19)

• Figure 1: LLM prompts features both system-level instructions which may include CoT instructions, personas, and other rules (orange), task-specific instructions which may include details and examples (blue), and the instance itself (green). Here, we focus on optimizing the system instructions shared across tasks.
• Figure 2: The Sprig pipeline where System Prompts are iteratively optimized through exploratory edits and promoted across iterations using combined benchmark to rank candidates.
• Figure 3: Average Score Improvement of all prompt optimization methods relative to the unoptimized setting, aggregated across LLMs. Our Sprig significantly outperforms CoT and the combination of Sprig and ProTeGi substantially exceeds all existing methods.
• Figure 4: Average score on the test set at each iteration when running Sprig. All three LLMs see significant improvements. Error bars show the variance in each beam.
• Figure 5: Z-scores by iteration for the number of components added of each type, showing which types were added more/less frequently than by chance; statistically significant rates are marked with $\times$.