Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RLAP: A Reinforcement Learning Enhanced Adaptive Planning Framework for Multi-step NLP Task Solving

Zepeng Ding, Dixuan Wang, Ziqin Luo, Guochao Jiang, Deqing Yang, Jiaqing Liang

TL;DR

This work addresses subtask ordering in multi-step NLP tasks by formulating the problem as a Markov decision process and embedding the LLM as the environment. A lightweight Actor model is trained with Deep Q-learning to estimate Q-values and guide adaptive subtask sequencing based on linguistic features, without fine-tuning the LLM. Across machine reading comprehension, information extraction, and sentence-level text completion, RLAP yields significant performance gains over fixed-order baselines and shows robustness to increased task complexity. The approach offers a practical, generalizable framework for improving multi-step NLP reasoning and planning with minimal dependency on task-specific LLM fine-tuning, with potential extensions to pre-training and multimodal settings.

Abstract

Multi-step planning has been widely employed to enhance the performance of large language models (LLMs) on downstream natural language processing (NLP) tasks, which decomposes the original task into multiple subtasks and guide LLMs to solve them sequentially without additional training. When addressing task instances, existing methods either preset the order of steps or attempt multiple paths at each step. However, these methods overlook instances' linguistic features and rely on the intrinsic planning capabilities of LLMs to evaluate intermediate feedback and then select subtasks, resulting in suboptimal outcomes. To better solve multi-step NLP tasks with LLMs, in this paper we propose a Reinforcement Learning enhanced Adaptive Planning framework (RLAP). In our framework, we model an NLP task as a Markov decision process (MDP) and employ an LLM directly into the environment. In particular, a lightweight Actor model is trained to estimate Q-values for natural language sequences consisting of states and actions through reinforcement learning. Therefore, during sequential planning, the linguistic features of each sequence in the MDP can be taken into account, and the Actor model interacts with the LLM to determine the optimal order of subtasks for each task instance. We apply RLAP on three different types of NLP tasks and conduct extensive experiments on multiple datasets to verify RLAP's effectiveness and robustness.

RLAP: A Reinforcement Learning Enhanced Adaptive Planning Framework for Multi-step NLP Task Solving

TL;DR

This work addresses subtask ordering in multi-step NLP tasks by formulating the problem as a Markov decision process and embedding the LLM as the environment. A lightweight Actor model is trained with Deep Q-learning to estimate Q-values and guide adaptive subtask sequencing based on linguistic features, without fine-tuning the LLM. Across machine reading comprehension, information extraction, and sentence-level text completion, RLAP yields significant performance gains over fixed-order baselines and shows robustness to increased task complexity. The approach offers a practical, generalizable framework for improving multi-step NLP reasoning and planning with minimal dependency on task-specific LLM fine-tuning, with potential extensions to pre-training and multimodal settings.

Abstract

Multi-step planning has been widely employed to enhance the performance of large language models (LLMs) on downstream natural language processing (NLP) tasks, which decomposes the original task into multiple subtasks and guide LLMs to solve them sequentially without additional training. When addressing task instances, existing methods either preset the order of steps or attempt multiple paths at each step. However, these methods overlook instances' linguistic features and rely on the intrinsic planning capabilities of LLMs to evaluate intermediate feedback and then select subtasks, resulting in suboptimal outcomes. To better solve multi-step NLP tasks with LLMs, in this paper we propose a Reinforcement Learning enhanced Adaptive Planning framework (RLAP). In our framework, we model an NLP task as a Markov decision process (MDP) and employ an LLM directly into the environment. In particular, a lightweight Actor model is trained to estimate Q-values for natural language sequences consisting of states and actions through reinforcement learning. Therefore, during sequential planning, the linguistic features of each sequence in the MDP can be taken into account, and the Actor model interacts with the LLM to determine the optimal order of subtasks for each task instance. We apply RLAP on three different types of NLP tasks and conduct extensive experiments on multiple datasets to verify RLAP's effectiveness and robustness.
Paper Structure (15 sections, 7 equations, 5 figures, 4 tables, 2 algorithms)

This paper contains 15 sections, 7 equations, 5 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-step Planning for LLMs
  4. Reinforcement Learning in NLP Tasks
  5. Framework of RLAP
  6. Preliminaries
  7. Environment Construction
  8. Training Process of Actor Model
  9. RLAP Applications for Different NLP Tasks
  10. Machine Reading Comprehension
  11. Information Extraction
  12. Sentence-level Text Completion
  13. Conclusion
  14. Dataset Introduction
  15. Experiment Details

Figures (5)

  • Figure 1: (a) For the same task instance, different action orders will lead to different results. (b) Overview of the proposed RLAP.
  • Figure 2: An illustration of RLAP in MRC tasks.
  • Figure 3: An example of RLAP in IE tasks.
  • Figure 4: An illustration of RLAP in STC tasks.
  • Figure 5: The X-axis represents the number of blanks per sample, and the numbers in the legend indicate the metric values on the entire test set.