Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Supervisory Prompt Training

Jean Ghislain Billa, Min Oh, Liang Du

TL;DR

SPT introduces a self-improving, dual-LLM framework in which a generator and a corrector iteratively refine prompts to boost performance and reduce hallucinations. It adds impact scores to quantify sentence-level contributions, guiding subsequent improvements and enabling the corrector to evolve its feedback. Across four benchmarks, SPT yields substantial gains—most notably a $28.3$-point GSM8K improvement for GPT-4—and demonstrates that refining prompts can rival traditional fine-tuning as a scalable alternative. The approach advances prompt-based optimization, offering a practical path to more reliable LLM behavior in high-stakes and diverse tasks.

Abstract

The performance of Large Language Models (LLMs) relies heavily on the quality of prompts, which are often manually engineered and task-specific, making them costly and non-scalable. We propose a novel approach, Supervisory Prompt Training (SPT). SPT automates the generation of highly effective prompts using a dual LLM system. In this system, one LLM, the generator, performs a task while the other, the corrector, provides feedback and generates improved prompts. In contrast to earlier techniques, both the generator and corrector collaboratively and continuously improve their prompts over time. We also introduce the concept of \textit{impact scores} to measure the sentence-level effectiveness of the prompts. Our method was tested on four benchmarks, testing the level of hallucinations in LLMs. Notably, we were able to increase the accuracy of GPT-4 on GSM8K from 65.8\% to 94.1\% (28.3\% increase). SPT advances LLMs by refining prompts to enhance performance and reduce hallucinations, offering an efficient and scalable alternative to traditional model fine-tuning.

Supervisory Prompt Training

TL;DR

SPT introduces a self-improving, dual-LLM framework in which a generator and a corrector iteratively refine prompts to boost performance and reduce hallucinations. It adds impact scores to quantify sentence-level contributions, guiding subsequent improvements and enabling the corrector to evolve its feedback. Across four benchmarks, SPT yields substantial gains—most notably a -point GSM8K improvement for GPT-4—and demonstrates that refining prompts can rival traditional fine-tuning as a scalable alternative. The approach advances prompt-based optimization, offering a practical path to more reliable LLM behavior in high-stakes and diverse tasks.

Abstract

The performance of Large Language Models (LLMs) relies heavily on the quality of prompts, which are often manually engineered and task-specific, making them costly and non-scalable. We propose a novel approach, Supervisory Prompt Training (SPT). SPT automates the generation of highly effective prompts using a dual LLM system. In this system, one LLM, the generator, performs a task while the other, the corrector, provides feedback and generates improved prompts. In contrast to earlier techniques, both the generator and corrector collaboratively and continuously improve their prompts over time. We also introduce the concept of \textit{impact scores} to measure the sentence-level effectiveness of the prompts. Our method was tested on four benchmarks, testing the level of hallucinations in LLMs. Notably, we were able to increase the accuracy of GPT-4 on GSM8K from 65.8\% to 94.1\% (28.3\% increase). SPT advances LLMs by refining prompts to enhance performance and reduce hallucinations, offering an efficient and scalable alternative to traditional model fine-tuning.
Paper Structure (25 sections, 17 figures, 3 tables)

This paper contains 25 sections, 17 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Continuous optimization
  4. Discrete optimization
  5. Proposed method
  6. Initial setup
  7. Iteratively updating p (SPT-p)
  8. Iteratively updating p and c (SPT-pc)
  9. Extended Methods
  10. Iteratively improving p and c with chain-of-thought reasoning (SPT-cot)
  11. Impact scores (SPT-imp)
  12. Experiments
  13. Problem
  14. Models
  15. Benchmarks
  16. ...and 10 more sections

Figures (17)

  • Figure 1: Flowchart illustrating the iterative process of improving a generator's meta-prompt. Pink circles represent the meta-prompts in the process, purple trapezes represent data, and orange rectangles represent LLMs.
  • Figure 2: Flowchart illustrating the iterative process of improving a corrector's meta-prompt.
  • Figure 3: Flowchart illustrating the iterative process of generating impact scores.
  • Figure A.1: Comparison of SPT to the generator's baseline and APO. SPT does consistently better than the baselines on TruthfulQA and GSM8K.
  • Figure A.2: GPT4 Generator's meta-prompt for GSM8K
  • ...and 12 more figures