Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PRACT: Optimizing Principled Reasoning and Acting of LLM Agent

Zhiwei Liu, Weiran Yao, Jianguo Zhang, Rithesh Murthy, Liangwei Yang, Zuxin Liu, Tian Lan, Ming Zhu, Juntao Tan, Shirley Kokane, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong

TL;DR

Experimental results across four environments demonstrate that the PRAct agent, leveraging the RPO framework, can effectively learn and apply action principles to enhance performance.

Abstract

We introduce the Principled Reasoning and Acting (PRAct) framework, a novel method for learning and enforcing action principles from trajectory data. Central to our approach is the use of text gradients from a reflection and optimization engine to derive these action principles. To adapt action principles to specific task requirements, we propose a new optimization framework, Reflective Principle Optimization (RPO). After execution, RPO employs a reflector to critique current action principles and an optimizer to update them accordingly. We develop the RPO framework under two scenarios: Reward-RPO, which uses environmental rewards for reflection, and Self-RPO, which conducts self-reflection without external rewards. Additionally, two RPO methods, RPO-Traj and RPO-Batch, is introduced to adapt to different settings. Experimental results across four environments demonstrate that the PRAct agent, leveraging the RPO framework, effectively learns and applies action principles to enhance performance.

PRACT: Optimizing Principled Reasoning and Acting of LLM Agent

TL;DR

Experimental results across four environments demonstrate that the PRAct agent, leveraging the RPO framework, can effectively learn and apply action principles to enhance performance.

Abstract

We introduce the Principled Reasoning and Acting (PRAct) framework, a novel method for learning and enforcing action principles from trajectory data. Central to our approach is the use of text gradients from a reflection and optimization engine to derive these action principles. To adapt action principles to specific task requirements, we propose a new optimization framework, Reflective Principle Optimization (RPO). After execution, RPO employs a reflector to critique current action principles and an optimizer to update them accordingly. We develop the RPO framework under two scenarios: Reward-RPO, which uses environmental rewards for reflection, and Self-RPO, which conducts self-reflection without external rewards. Additionally, two RPO methods, RPO-Traj and RPO-Batch, is introduced to adapt to different settings. Experimental results across four environments demonstrate that the PRAct agent, leveraging the RPO framework, effectively learns and applies action principles to enhance performance.

Paper Structure

This paper contains 14 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. PRAct: Optimizing Principled Reasoning and Acting
  3. Formulation
  4. Reflective Principle Optimization (RPO)
  5. Execution
  6. Self-Reflection
  7. Optimization
  8. Experiment
  9. Experiment Setup
  10. Optimization setup
  11. Experiment Results
  12. Overall Performance.
  13. Optimization Curve.
  14. Conclusion

Figures (3)

  • Figure 1: Comparison of ReAct and PRAct agents.
  • Figure 2: PRAct and RPO overview. Each iteration three stages: execution, reflection and optimization. During execution, an agent executes tasks with previous principles. The trajectories are saved. Then, the agent reflects on those tasks executions. Finally, the agent leverages those self-reflection results to optimize the principle.
  • Figure 3: Training curves in Webshop and Academia with different LLMs and data splits. The reported scores are the average across 5 random seeds.