Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GRL-Prompt: Towards Knowledge Graph based Prompt Optimization via Reinforcement Learning

Yuze Liu, Tingjie Liu, Tiehua Zhang, Youhua Xia, Jinze Wang, Zhishu Shen, Jiong Jin, Fei Richard Yu

TL;DR

A novel LLMs-agnostic framework for prompt optimization, namely GRL-Prompt, which aims to automatically construct optimal prompts via reinforcement learning (RL) in an end-to-end manner and outperforms recent state-of-the-art methods.

Abstract

Large language models (LLMs) have demonstrated impressive success in a wide range of natural language processing (NLP) tasks due to their extensive general knowledge of the world. Recent works discovered that the performance of LLMs is heavily dependent on the input prompt. However, prompt engineering is usually done manually in a trial-and-error fashion, which can be labor-intensive and challenging in order to find the optimal prompts. To address these problems and unleash the utmost potential of LLMs, we propose a novel LLMs-agnostic framework for prompt optimization, namely GRL-Prompt, which aims to automatically construct optimal prompts via reinforcement learning (RL) in an end-to-end manner. To provide structured action/state representation for optimizing prompts, we construct a knowledge graph (KG) that better encodes the correlation between the user query and candidate in-context examples. Furthermore, a policy network is formulated to generate the optimal action by selecting a set of in-context examples in a rewardable order to construct the prompt. Additionally, the embedding-based reward shaping is utilized to stabilize the RL training process. The experimental results show that GRL-Prompt outperforms recent state-of-the-art methods, achieving an average increase of 0.10 in ROUGE-1, 0.07 in ROUGE-2, 0.07 in ROUGE-L, and 0.05 in BLEU.

GRL-Prompt: Towards Knowledge Graph based Prompt Optimization via Reinforcement Learning

TL;DR

A novel LLMs-agnostic framework for prompt optimization, namely GRL-Prompt, which aims to automatically construct optimal prompts via reinforcement learning (RL) in an end-to-end manner and outperforms recent state-of-the-art methods.

Abstract

Large language models (LLMs) have demonstrated impressive success in a wide range of natural language processing (NLP) tasks due to their extensive general knowledge of the world. Recent works discovered that the performance of LLMs is heavily dependent on the input prompt. However, prompt engineering is usually done manually in a trial-and-error fashion, which can be labor-intensive and challenging in order to find the optimal prompts. To address these problems and unleash the utmost potential of LLMs, we propose a novel LLMs-agnostic framework for prompt optimization, namely GRL-Prompt, which aims to automatically construct optimal prompts via reinforcement learning (RL) in an end-to-end manner. To provide structured action/state representation for optimizing prompts, we construct a knowledge graph (KG) that better encodes the correlation between the user query and candidate in-context examples. Furthermore, a policy network is formulated to generate the optimal action by selecting a set of in-context examples in a rewardable order to construct the prompt. Additionally, the embedding-based reward shaping is utilized to stabilize the RL training process. The experimental results show that GRL-Prompt outperforms recent state-of-the-art methods, achieving an average increase of 0.10 in ROUGE-1, 0.07 in ROUGE-2, 0.07 in ROUGE-L, and 0.05 in BLEU.

Paper Structure

This paper contains 27 sections, 10 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary and Problem Definition
  4. Methodology
  5. Knowledge Graph for Prompt Construction
  6. Knowledge Graph Construction
  7. Knowledge Graph Learning
  8. RL-based Prompt Optimization
  9. Policy Network
  10. Reward Design
  11. Training Process
  12. Experiment
  13. Experimental Setup
  14. Dataset
  15. Baselines and Configurations
  16. ...and 12 more sections

Figures (12)

  • Figure 1: An overview of GRL-Prompt. The circle and triangle represent the candidate nodes and the query node, respectively. The green arrow lines denote candidate-to-candidate edges, the orange arrow lines indicate query-to-candidate edges, and the blue arrow lines represent candidate-to-query edges in the knowledge graph.
  • Figure 1: The demonstration of the prompt template
  • Figure 2: Results of training loss on different datasets.
  • Figure 2: The trend of training loss across different LLMs on two datasets.
  • Figure 3: Sensitivity analysis on HGT layer.
  • ...and 7 more figures