Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMs

Zhuo Li, Yuhao Du, Jinpeng Hu, Xiang Wan, Anningzhe Gao

TL;DR

This work introduces a self-instructed in-context learning framework that empowers LLMs to deliver more effective responses by generating reliable derived prompts to construct informative contextual environments and significantly enhances LLMs' ability to deliver more effective responses, including Black-Box models such as GPT-4.

Abstract

Large language models (LLMs) have shown success in generating high-quality responses. In order to achieve better alignment with LLMs with human preference, various works are proposed based on specific optimization process, which, however, is not suitable to Black-Box LLMs like GPT-4, due to inaccessible parameters. In Black-Box LLMs case, their performance is highly dependent on the quality of the provided prompts. Existing methods to enhance response quality often involve a prompt refinement model, yet these approaches potentially suffer from semantic inconsistencies between the refined and original prompts, and typically overlook the relationship between them. To address these challenges, we introduce a self-instructed in-context learning framework that empowers LLMs to deliver more effective responses by generating reliable derived prompts to construct informative contextual environments. Our approach incorporates a self-instructed reinforcement learning mechanism, enabling direct interaction with the response model during derived prompt generation for better alignment. We then formulate querying as an in-context learning task, using responses from LLMs combined with the derived prompts to establish a contextual demonstration for the original prompt. This strategy ensures alignment with the original query, reduces discrepancies from refined prompts, and maximizes the LLMs' in-context learning capability. Extensive experiments demonstrate that the proposed method not only generates more reliable derived prompts but also significantly enhances LLMs' ability to deliver more effective responses, including Black-Box models such as GPT-4.

Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMs

TL;DR

This work introduces a self-instructed in-context learning framework that empowers LLMs to deliver more effective responses by generating reliable derived prompts to construct informative contextual environments and significantly enhances LLMs' ability to deliver more effective responses, including Black-Box models such as GPT-4.

Abstract

Large language models (LLMs) have shown success in generating high-quality responses. In order to achieve better alignment with LLMs with human preference, various works are proposed based on specific optimization process, which, however, is not suitable to Black-Box LLMs like GPT-4, due to inaccessible parameters. In Black-Box LLMs case, their performance is highly dependent on the quality of the provided prompts. Existing methods to enhance response quality often involve a prompt refinement model, yet these approaches potentially suffer from semantic inconsistencies between the refined and original prompts, and typically overlook the relationship between them. To address these challenges, we introduce a self-instructed in-context learning framework that empowers LLMs to deliver more effective responses by generating reliable derived prompts to construct informative contextual environments. Our approach incorporates a self-instructed reinforcement learning mechanism, enabling direct interaction with the response model during derived prompt generation for better alignment. We then formulate querying as an in-context learning task, using responses from LLMs combined with the derived prompts to establish a contextual demonstration for the original prompt. This strategy ensures alignment with the original query, reduces discrepancies from refined prompts, and maximizes the LLMs' in-context learning capability. Extensive experiments demonstrate that the proposed method not only generates more reliable derived prompts but also significantly enhances LLMs' ability to deliver more effective responses, including Black-Box models such as GPT-4.
Paper Structure (34 sections, 5 equations, 5 figures, 5 tables, 2 algorithms)

This paper contains 34 sections, 5 equations, 5 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Supervised Fine-Tuning
  4. Reinforcement Learning from Human Feedback
  5. Motivation
  6. Methodology
  7. Task Formulation
  8. Self-instructed RL for Derived Prompt Generation Model
  9. Intent-consistency Oriented In-context Query Framework for Inference
  10. Implementation Details
  11. Reward Model $\mathcal{R}$.
  12. Response Model $\mathcal{M}$.
  13. Experiments
  14. Experimental Setup
  15. Datasets.
  16. ...and 19 more sections

Figures (5)

  • Figure 1: (a) Traditional methods directly replace the original prompt with a refined one, potentially risking semantic inconsistencies and ineffective responses. (b) Our method uses a derived prompt to create an in-context demonstration, ensuring high-quality responses while maintaining the integrity of the original prompt.
  • Figure 2: Win rates comparison in Vicuna Eval among w/o training, SFT and RL. Based on Llama3-8B for generation of derived prompt.
  • Figure 3: Detailed cases study. We compare quality of different types of responses by querying GPT-4.
  • Figure 4:
  • Figure 5: