Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learning

Wenjin Liu, Haoran Luo, Xueyuan Lin, Haoming Liu, Tiesunlong Shen, Jiapu Wang, Rui Mao, Erik Cambria

TL;DR

Prompt-R1 presents a collaborative automatic prompting framework in which a small-scale LLM acts as an agent that learns to prompt a large-scale LLM through end-to-end reinforcement learning. By implementing a dual-constrained reward and a plug-and-play architecture, the approach enables multi-turn interactions that improve correctness and generation quality without fine-tuning large models. Empirical results across multi-hop reasoning, QA, math computation, and text generation demonstrate consistent improvements over strong baselines and good generalization to out-of-distribution tasks, including transferring to different LLM environments. The work highlights the potential of cross-LLM collaboration to reduce prompting costs while boosting robustness and adaptability in complex reasoning tasks.

Abstract

Recently, advanced large language models (LLMs) have emerged at an increasingly rapid pace. However, when faced with complex problems, most users are often unable to provide accurate and effective prompts to interact with LLMs, thus limiting the performance of LLMs. To address this challenge, we propose Prompt-R1, an end-to-end reinforcement learning framework that uses a small-scale LLM to collaborate with large-scale LLMs, replacing user interaction to solve problems better. This collaboration is cast as a multi-turn prompt interaction, where the small-scale LLM thinks and generates prompts, and the large-scale LLM performs complex reasoning. A dual-constrained reward is designed to optimize for correctness, generation quality, and reasoning accuracy. Prompt-R1 provides a plug-and-play framework that supports both inference and training with various large-scale LLMs. Experiments on multiple public datasets show that Prompt-R1 significantly outperforms baseline models across tasks. Our code is publicly available at https://github.com/QwenQKing/Prompt-R1.

Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learning

TL;DR

Prompt-R1 presents a collaborative automatic prompting framework in which a small-scale LLM acts as an agent that learns to prompt a large-scale LLM through end-to-end reinforcement learning. By implementing a dual-constrained reward and a plug-and-play architecture, the approach enables multi-turn interactions that improve correctness and generation quality without fine-tuning large models. Empirical results across multi-hop reasoning, QA, math computation, and text generation demonstrate consistent improvements over strong baselines and good generalization to out-of-distribution tasks, including transferring to different LLM environments. The work highlights the potential of cross-LLM collaboration to reduce prompting costs while boosting robustness and adaptability in complex reasoning tasks.

Abstract

Recently, advanced large language models (LLMs) have emerged at an increasingly rapid pace. However, when faced with complex problems, most users are often unable to provide accurate and effective prompts to interact with LLMs, thus limiting the performance of LLMs. To address this challenge, we propose Prompt-R1, an end-to-end reinforcement learning framework that uses a small-scale LLM to collaborate with large-scale LLMs, replacing user interaction to solve problems better. This collaboration is cast as a multi-turn prompt interaction, where the small-scale LLM thinks and generates prompts, and the large-scale LLM performs complex reasoning. A dual-constrained reward is designed to optimize for correctness, generation quality, and reasoning accuracy. Prompt-R1 provides a plug-and-play framework that supports both inference and training with various large-scale LLMs. Experiments on multiple public datasets show that Prompt-R1 significantly outperforms baseline models across tasks. Our code is publicly available at https://github.com/QwenQKing/Prompt-R1.

Paper Structure

This paper contains 32 sections, 58 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method: Prompt-R1
  4. Multi-Turn Interaction of Agent and LLM
  5. Double-constrained RL Optimization
  6. Efficient Training and Inference
  7. Experiments
  8. Experimental Setup
  9. Main Results (RQ1)
  10. Generalization Results (RQ2)
  11. Work with Other LLMs (RQ3)
  12. Ablation study (RQ4)
  13. Different environment Training (RQ5)
  14. Conclusion
  15. Example Appendix
  16. ...and 17 more sections

Figures (9)

  • Figure 1: An example of Prompt-R1 agent works with a large-scale LLM. The agent obtains the correct answer by interacting with the large-scale LLM step by step.
  • Figure 2: Comparison of different methods for improving the performance of LLMs: human-LLM interaction, prompt engineering, fine-tuning optimization, and our collaborative automatic prompting interaction framework.
  • Figure 3: An overview of the Prompt-R1 framework. A small-scale LLM (as agent) interacts with a large-scale LLM (as environment) to answer questions through multi-turn prompts. The large-scale LLM is plug-and-play.
  • Figure 4: The performance of six LLMs without Prompt-R1 Agent is compared to that with Prompt-R1 on eight datasets, using appropriate metrics. F1: multi-hop reasoning (2Wiki, Hotpot) and standard QA (MusiQue, PopQA); EM: mathematical computation (GSM8K, DAPO); and SSim: text generation (BookSum, W.P.).
  • Figure 5: Comparison of the average values of the four EM, F1, and SSim scores for the selected six LLM baselines on OOD datasets, as well as the average of these scores for these LLMs using the Prompt-R1 agent.
  • ...and 4 more figures

Theorems & Definitions (6)

  • proof
  • proof
  • proof
  • proof
  • proof
  • proof