Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts

Ruochen Wang, Sohyun An, Minhao Cheng, Tianyi Zhou, Sung Ju Hwang, Cho-Jui Hsieh

TL;DR

This paper tackles the limited problem-space coverage of single-instruction prompt engineering for LLMs by introducing Mixture-of-Prompts (MoP), a Mixture-of-Experts-inspired framework where each region of the problem space is governed by a dedicated expert prompt (an instruction plus demos). MoP uses a two-phase process: (1) Demo Assignment, clustering demos in embedding space to form region-specific experts, guided by the kernel-regression view of in-context learning; and (2) Instruction Assignment, a Region-Based Joint Search (RBJS) to jointly optimize the best instruction per expert given its demos. Inference routes queries to the closest expert in embedding space, and each expert uses its region-specific prompt to produce outputs, leading to substantial performance gains. Empirical results across Instruction Induction, Super Natural Instructions, and BIG-Bench-Hard show MoP achieving an average win rate of 81% against baselines and surpassing single-instruction methods by large margins, demonstrating the value of region-specific prompt design for broad task spaces. The work highlights practical considerations for prompt optimization, including clustering choices, embedding models, and region-aware evaluation, making MoP a scalable and adaptable approach for automated prompt construction with real-world impact on LLM generalization and efficiency.

Abstract

Large Language Models (LLMs) exhibit strong generalization capabilities to novel tasks when prompted with language instructions and in-context demos. Since this ability sensitively depends on the quality of prompts, various methods have been explored to automate the instruction design. While these methods demonstrated promising results, they also restricted the searched prompt to one instruction. Such simplification significantly limits their capacity, as a single demo-free instruction might not be able to cover the entire complex problem space of the targeted task. To alleviate this issue, we adopt the Mixture-of-Expert paradigm and divide the problem space into a set of sub-regions; Each sub-region is governed by a specialized expert, equipped with both an instruction and a set of demos. A two-phase process is developed to construct the specialized expert for each region: (1) demo assignment: Inspired by the theoretical connection between in-context learning and kernel regression, we group demos into experts based on their semantic similarity; (2) instruction assignment: A region-based joint search of an instruction per expert complements the demos assigned to it, yielding a synergistic effect. The resulting method, codenamed Mixture-of-Prompts (MoP), achieves an average win rate of 81% against prior arts across several major benchmarks.

One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts

TL;DR

This paper tackles the limited problem-space coverage of single-instruction prompt engineering for LLMs by introducing Mixture-of-Prompts (MoP), a Mixture-of-Experts-inspired framework where each region of the problem space is governed by a dedicated expert prompt (an instruction plus demos). MoP uses a two-phase process: (1) Demo Assignment, clustering demos in embedding space to form region-specific experts, guided by the kernel-regression view of in-context learning; and (2) Instruction Assignment, a Region-Based Joint Search (RBJS) to jointly optimize the best instruction per expert given its demos. Inference routes queries to the closest expert in embedding space, and each expert uses its region-specific prompt to produce outputs, leading to substantial performance gains. Empirical results across Instruction Induction, Super Natural Instructions, and BIG-Bench-Hard show MoP achieving an average win rate of 81% against baselines and surpassing single-instruction methods by large margins, demonstrating the value of region-specific prompt design for broad task spaces. The work highlights practical considerations for prompt optimization, including clustering choices, embedding models, and region-aware evaluation, making MoP a scalable and adaptable approach for automated prompt construction with real-world impact on LLM generalization and efficiency.

Abstract

Large Language Models (LLMs) exhibit strong generalization capabilities to novel tasks when prompted with language instructions and in-context demos. Since this ability sensitively depends on the quality of prompts, various methods have been explored to automate the instruction design. While these methods demonstrated promising results, they also restricted the searched prompt to one instruction. Such simplification significantly limits their capacity, as a single demo-free instruction might not be able to cover the entire complex problem space of the targeted task. To alleviate this issue, we adopt the Mixture-of-Expert paradigm and divide the problem space into a set of sub-regions; Each sub-region is governed by a specialized expert, equipped with both an instruction and a set of demos. A two-phase process is developed to construct the specialized expert for each region: (1) demo assignment: Inspired by the theoretical connection between in-context learning and kernel regression, we group demos into experts based on their semantic similarity; (2) instruction assignment: A region-based joint search of an instruction per expert complements the demos assigned to it, yielding a synergistic effect. The resulting method, codenamed Mixture-of-Prompts (MoP), achieves an average win rate of 81% against prior arts across several major benchmarks.
Paper Structure (65 sections, 1 theorem, 13 equations, 6 figures, 12 tables, 2 algorithms)

This paper contains 65 sections, 1 theorem, 13 equations, 6 figures, 12 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Prompt optimization for language generation.
  4. Mixture of Experts Paradigm.
  5. Preliminaries
  6. Terminology.
  7. Prompt Optimization.
  8. APE - Automatic Prompt Engineering.
  9. Limitations of APE.
  10. MoP: Mixture-of-Prompts
  11. Framework Overview
  12. Mixture-of-Expert for prompt optimization.
  13. Demo Assignment
  14. LLM learns from demos via Kernel Regression in the embedding space.
  15. Clustering demos to each expert based on their semantic similarity.
  16. ...and 50 more sections

Key Result

Theorem 4.1

Let $\{(x_i, y_i)\}_{i=1}^n$ denote the demos used in the prompt; Let $K$ define a kernel function that measures the semantic similarity between two data points, which can be represented as $K(x_i, x_j)= \phi(x_i)^T \phi(x_j)$ with some embedding space $\phi(\cdot)$. Then the output of LLM, $P(y | [

Figures (6)

  • Figure 1: Illustration of MoP. We adopt the MoE paradigm and divide the problem space into a set of sub-regions. Each sub-region is governed by a specialized expert, equipped with both an instruction and a set of demos. A two-phase process is developed to construct the specialized expert for each region: (1) demo assignment: Inspired by the theoretical connection between ICL and kernel regression, we group demos into experts based on their semantic similarity; (2) instruction assignment: A region-based joint search (RBJS) of an instruction per expert complements the demos assigned to it, yielding a synergistic effect. During inference, each new query is routed to its closest expert in the embedding space and the assigned expert then utilizes its prompt (instruction + demos) to make the final prediction.
  • Figure 2: Analysis.(a) There exist underlying patterns in the data distribution, and demos with semantically similar meanings are grouped closely. The circle, triangle, and star shapes represent training, routed validation, and routed test demos, respectively. (b) Each expert has distinct task-solving ability for each input. (c) A single instruction is insufficient for all experts, highlighting the need for distinct synergistic instructions for each expert. (d) Performance of instructions evaluated under local data distribution for each expert (i.e. subsets routed to each expert) is not aligned with the full data; This motivates performing region-based evaluation during joint search (RBJS).
  • Figure 3: Main results. We validate MoP across three major prompt optimization benchmarks. MoP achieves an average performance of 52.73% outperforming the average performance of 41.39% / 39.87% achieved by APE+Demos / IZ+Demos in these results.
  • Figure 4: Win rate matrices. We compare the pairwise win rate of all methods on Instruction Induction (a), SuperNI (b), and BIG-Bench-Hard (c). Our method achieves the best win rate against all six baselines across various benchmarks. The average win rate of MoP across all benchmarks is 81%.
  • Figure 5: Ablation study on different number of demos. We measure the task performance of each method across different numbers of demos. Here, $N$ on the x-axis represents the total number of training demos.
  • ...and 1 more figures

Theorems & Definitions (1)

  • Theorem 4.1