Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AGILE: A Novel Reinforcement Learning Framework of LLM Agents

Peiyuan Feng, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, Hang Li

Abstract

We introduce a novel reinforcement learning framework of LLM agents named AGILE (AGent that Interacts and Learns from Environments) designed to perform complex conversational tasks with users, leveraging LLMs, memory, tools, and interactions with experts. The agent possesses capabilities beyond conversation, including reflection, tool usage, and expert consultation. We formulate the construction of such an LLM agent as a reinforcement learning (RL) problem, in which the LLM serves as the policy model. We fine-tune the LLM using labeled data of actions and the PPO algorithm. We focus on question answering and release a dataset for agents called ProductQA, comprising challenging questions in online shopping. Our extensive experiments on ProductQA, MedMCQA and HotPotQA show that AGILE agents based on 7B and 13B LLMs trained with PPO can outperform GPT-4 agents. Our ablation study highlights the indispensability of memory, tools, consultation, reflection, and reinforcement learning in achieving the agent's strong performance. Datasets and code are available at https://github.com/bytarnish/AGILE.

AGILE: A Novel Reinforcement Learning Framework of LLM Agents

Abstract

We introduce a novel reinforcement learning framework of LLM agents named AGILE (AGent that Interacts and Learns from Environments) designed to perform complex conversational tasks with users, leveraging LLMs, memory, tools, and interactions with experts. The agent possesses capabilities beyond conversation, including reflection, tool usage, and expert consultation. We formulate the construction of such an LLM agent as a reinforcement learning (RL) problem, in which the LLM serves as the policy model. We fine-tune the LLM using labeled data of actions and the PPO algorithm. We focus on question answering and release a dataset for agents called ProductQA, comprising challenging questions in online shopping. Our extensive experiments on ProductQA, MedMCQA and HotPotQA show that AGILE agents based on 7B and 13B LLMs trained with PPO can outperform GPT-4 agents. Our ablation study highlights the indispensability of memory, tools, consultation, reflection, and reinforcement learning in achieving the agent's strong performance. Datasets and code are available at https://github.com/bytarnish/AGILE.
Paper Structure (63 sections, 8 equations, 15 figures, 19 tables, 1 algorithm)

This paper contains 63 sections, 8 equations, 15 figures, 19 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. RL formulation of agent
  4. Policy learning
  5. Interaction with human experts
  6. The ProductQA dataset
  7. Product collection
  8. QA collection
  9. Fact-QA
  10. Search-QA
  11. Reasoning-QA
  12. Experiments
  13. Experimental setting
  14. Dataset
  15. Agent definition
  16. ...and 48 more sections

Figures (15)

  • Figure 1: (a) Architecture of our agent system, including LLM, memory, tools, and executor. (b) A running example of AGILE in a customer service QA environment. The tokens (actions) generated by the LLM are in orange color and the tokens appended by the executor are in blue color.
  • Figure 2: Accuracy and advice rate over the following 200 sessions ($c = 0.3$).
  • Figure 3: Advice rate, accuracy along with seeking advice cost $c$ on ProductQA.
  • Figure 4: Advice rate over the following 200 sessions on ProductQA ($c = 0.3$).
  • Figure 5: Reward and value function loss curves during the PPO training process on ProductQA.
  • ...and 10 more figures