Empowering Large Language Model Agents through Action Learning

TL;DR

This work introduces a framework LearnAct with an iterative learning strategy to create and improve actions in the form of Python functions that markedly improves agent performance for the type of task, highlighting the importance of experiential action learning in the development of more intelligent LLM agents.

Abstract

Large Language Model (LLM) Agents have recently garnered increasing interest yet they are limited in their ability to learn from trial and error, a key element of intelligent behavior. In this work, we argue that the capacity to learn new actions from experience is fundamental to the advancement of learning in LLM agents. While humans naturally expand their action spaces and develop skills through experiential learning, LLM agents typically operate within fixed action spaces, limiting their potential for growth. To address these challenges, our study explores open-action learning for language agents. We introduce a framework LearnAct with an iterative learning strategy to create and improve actions in the form of Python functions. In each iteration, LLM revises and updates the currently available actions based on the errors identified in unsuccessful training tasks, thereby enhancing action effectiveness. Our experimental evaluations across Robotic Planning and Alfworld environments reveal that after learning on a few training task instances, our approach to open-action learning markedly improves agent performance for the type of task (by 32 percent in AlfWorld compared to ReAct+Reflexion, for instance) highlighting the importance of experiential action learning in the development of more intelligent LLM agents.

Figures (10)

• Figure 1: Illustration of the training and test stage of LearnAct: Left: During the training stage, LearnAct expands the action space by first creating actions and then optimizing them based on the execution feedback. Right: The test stage uses learned action space to facilitate sequential decision-making. The prompting format follows the Act yao2023react agent.
• Figure 2: Left: During the learning stage, action usage by agent and action optimization are repeatedly executed. The improved action is evaluated on the training instances, identifying the failed case for the subsequent learning step. Actions are improved through either updating the functions or writing notes. Multiple samples are produced during the learning, and upon evaluation against training instances, the optimal one is selected for the next iteration. Right: Case example of action updating and note writing. The action update addresses previous shortcomings by refining functions for improved issue resolution. Conversely, note writing advises agents on proper action usage. LLMs have the freedom to select from two learning options.
• Figure 3: Left: The frequency of use and accuracy of learned actions before and after learning. Post-learning, there is a marked increase in the action usage frequency and accuracy, indicating their enhanced reliability and utility for the agent. Right: The performance with different maximum learning iteration steps. The performance varies with different maximum learning iteration steps. LearnAct's performance notably improves with the application of learning, particularly at step two. Although ReAct+Reflexion also shows improvement, its progress is less significant and stable.
• Figure 4: Prompt for action creation.
• Figure 5: Prompt for action description generation.