Boosting Instruction Following at Scale

TL;DR

Instruction-following reliability in LLMs degrades as prompt instructions scale. The authors introduce Instruction Boosting as a post-generation refinement, and ScaledIF as a benchmark to study up to ten instructions per sample, along with a soft-conflict based conflict-score to diagnose instruction interactions. Across multiple open models, boosting yields up to 7 percentage-point gains for two instructions and up to 4 points for ten instructions, while soft conflicts explain part of the degradation. The work provides practical tooling and a diagnostic framework to improve prompt design and control in real-world LLM applications.

Abstract

A typical approach developers follow to influence an LLM's behavior in an application is through careful manipulation of the prompt, such as by adding or modifying instructions. However, merely adding more instructions provides little assurance that they will actually be followed. We introduce Instruction Boosting as a post-generation method to increase the reliability of LLM prompt instructions. We show that Instruction Boosting improves the instruction following rate by up to 7 points for two instructions and up to 4 points for ten instructions. To demonstrate these results we introduce SCALEDIF, a benchmark with a scaled instruction volume of up to ten instructions per data sample. We also present an analysis of the commonly observed trend that performance degrades as more instructions are added. We show that an important factor contributing to this trend is the degree of tension and conflict that arises as the number of instructions is increased. We contribute a quantitative conflict scoring tool that explains the observed performance trends and provides feedback to developers on the impact that additional prompt instructions have on a model's performance.

Figures (7)

• Figure 1: Overview of the instruction boosting approach.
• Figure 2: Number of instructions in ScaledIF across eight instruction categories. The contribution of individual instruction classes in each category is distinguished by color.
• Figure 3: Initial IF rate (solid lines) and Best-of-N boosting performance (dashed lines).
• Figure 4: Instruction following rate achieved by Llama 70b for different boosting strategies.
• Figure 5: Costs (completion tokens in Flops) and IF rate of each strategy for 10 instructions achieved by Llama-70B.