Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAS: Data-Efficient Plug-and-Play Prompt Augmentation System

Miao Zheng, Hao Liang, Fan Yang, Haoze Sun, Tianpeng Li, Lingchu Xiong, Yan Zhang, Youzhen Wu, Kun Li, Yanjun Shen, Mingan Lin, Tao Zhang, Guosheng Dong, Yujing Qiao, Kun Fang, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou

TL;DR

PAS addresses the data inefficiency and limited plug-and-play flexibility of automatic prompt engineering for LLMs. It builds a curated prompt complementary dataset of ~9,000 high-quality (prompt, complementary prompt) pairs via automated selection, generation, and regeneration, and fine-tunes a prompt-complementary model M_p that augments user prompts at inference time. PAS achieves state-of-the-art performance across multiple benchmarks and model families with as few as 9k training pairs, outperforming the previous SoTA BPO by about 6.09 points and the baseline by about 8 points, with strong human-evaluation results and robust ablations. The approach is model-agnostic, task-agnostic, and data-efficient, enabling plug-and-play enhancement of LLMs without extra human labeling or retraining.

Abstract

In recent years, the rise of Large Language Models (LLMs) has spurred a growing demand for plug-and-play AI systems. Among the various AI techniques, prompt engineering stands out as particularly significant. However, users often face challenges in writing prompts due to the steep learning curve and significant time investment, and existing automatic prompt engineering (APE) models can be difficult to use. To address this issue, we propose PAS, an LLM-based plug-and-play APE system. PAS utilizes LLMs trained on high-quality, automatically generated prompt complementary datasets, resulting in exceptional performance. In comprehensive benchmarks, PAS achieves state-of-the-art (SoTA) results compared to previous APE models, with an average improvement of 6.09 points. Moreover, PAS is highly efficient, achieving SoTA performance with only 9000 data points. Additionally, PAS can autonomously generate prompt augmentation data without requiring additional human labor. Its flexibility also allows it to be compatible with all existing LLMs and applicable to a wide range of tasks. PAS excels in human evaluations, underscoring its suitability as a plug-in for users. This combination of high performance, efficiency, and flexibility makes PAS a valuable system for enhancing the usability and effectiveness of LLMs through improved prompt engineering.

PAS: Data-Efficient Plug-and-Play Prompt Augmentation System

TL;DR

PAS addresses the data inefficiency and limited plug-and-play flexibility of automatic prompt engineering for LLMs. It builds a curated prompt complementary dataset of ~9,000 high-quality (prompt, complementary prompt) pairs via automated selection, generation, and regeneration, and fine-tunes a prompt-complementary model M_p that augments user prompts at inference time. PAS achieves state-of-the-art performance across multiple benchmarks and model families with as few as 9k training pairs, outperforming the previous SoTA BPO by about 6.09 points and the baseline by about 8 points, with strong human-evaluation results and robust ablations. The approach is model-agnostic, task-agnostic, and data-efficient, enabling plug-and-play enhancement of LLMs without extra human labeling or retraining.

Abstract

In recent years, the rise of Large Language Models (LLMs) has spurred a growing demand for plug-and-play AI systems. Among the various AI techniques, prompt engineering stands out as particularly significant. However, users often face challenges in writing prompts due to the steep learning curve and significant time investment, and existing automatic prompt engineering (APE) models can be difficult to use. To address this issue, we propose PAS, an LLM-based plug-and-play APE system. PAS utilizes LLMs trained on high-quality, automatically generated prompt complementary datasets, resulting in exceptional performance. In comprehensive benchmarks, PAS achieves state-of-the-art (SoTA) results compared to previous APE models, with an average improvement of 6.09 points. Moreover, PAS is highly efficient, achieving SoTA performance with only 9000 data points. Additionally, PAS can autonomously generate prompt augmentation data without requiring additional human labor. Its flexibility also allows it to be compatible with all existing LLMs and applicable to a wide range of tasks. PAS excels in human evaluations, underscoring its suitability as a plug-in for users. This combination of high performance, efficiency, and flexibility makes PAS a valuable system for enhancing the usability and effectiveness of LLMs through improved prompt engineering.
Paper Structure (50 sections, 7 equations, 12 figures, 6 tables, 1 algorithm)

This paper contains 50 sections, 7 equations, 12 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automatic Prompt Engineering
  4. Plug-and-Play Systems
  5. Data Quality and Data Selection
  6. Data Quality
  7. Data Selection
  8. Method
  9. Prompts Data Collection
  10. Deduplication
  11. Quality Selection
  12. Classification
  13. Prompts Complementary Dataset
  14. Data Generation
  15. Data Selection and Regeneration
  16. ...and 35 more sections

Figures (12)

  • Figure 1: We first present the pipeline of the PAS in (a). PAS takes user prompts, enhances them, and then inputs the augmented prompts into LLMs. As illustrated in (b), PAS significantly improves responses across all categories in human evaluation.
  • Figure 2: Case Study 1, Red text is the complementary prompt generated by PAS. We can see PAS can give complementary prompt to avoid logic traps.
  • Figure 3: Pipeline for selecting prompt data and generating complementary prompt data.
  • Figure 4: Complementary Dataset Generation Prompt
  • Figure 5: Data Selection and Regeneration Prompt
  • ...and 7 more figures