Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning

Shengjie Sun, Runze Liu, Jiafei Lyu, Jing-Wen Yang, Liangpeng Zhang, Xiu Li

TL;DR

CARD, a LLM-driven Reward Design framework that iteratively generates and improves reward function code, includes a Coder that generates and verifies the code, while a Evaluator provides dynamic feedback to guide the Coder in improving the code, eliminating the need for human feedback.

Abstract

Large Language Models (LLMs) have shown significant potential in designing reward functions for Reinforcement Learning (RL) tasks. However, obtaining high-quality reward code often involves human intervention, numerous LLM queries, or repetitive RL training. To address these issues, we propose CARD, a LLM-driven Reward Design framework that iteratively generates and improves reward function code. Specifically, CARD includes a Coder that generates and verifies the code, while a Evaluator provides dynamic feedback to guide the Coder in improving the code, eliminating the need for human feedback. In addition to process feedback and trajectory feedback, we introduce Trajectory Preference Evaluation (TPE), which evaluates the current reward function based on trajectory preferences. If the code fails the TPE, the Evaluator provides preference feedback, avoiding RL training at every iteration and making the reward function better aligned with the task objective. Empirical results on Meta-World and ManiSkill2 demonstrate that our method achieves an effective balance between task performance and token efficiency, outperforming or matching the baselines across all tasks. On 10 out of 12 tasks, CARD shows better or comparable performance to policies trained with expert-designed rewards, and our method even surpasses the oracle on 3 tasks.

A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning

TL;DR

CARD, a LLM-driven Reward Design framework that iteratively generates and improves reward function code, includes a Coder that generates and verifies the code, while a Evaluator provides dynamic feedback to guide the Coder in improving the code, eliminating the need for human feedback.

Abstract

Large Language Models (LLMs) have shown significant potential in designing reward functions for Reinforcement Learning (RL) tasks. However, obtaining high-quality reward code often involves human intervention, numerous LLM queries, or repetitive RL training. To address these issues, we propose CARD, a LLM-driven Reward Design framework that iteratively generates and improves reward function code. Specifically, CARD includes a Coder that generates and verifies the code, while a Evaluator provides dynamic feedback to guide the Coder in improving the code, eliminating the need for human feedback. In addition to process feedback and trajectory feedback, we introduce Trajectory Preference Evaluation (TPE), which evaluates the current reward function based on trajectory preferences. If the code fails the TPE, the Evaluator provides preference feedback, avoiding RL training at every iteration and making the reward function better aligned with the task objective. Empirical results on Meta-World and ManiSkill2 demonstrate that our method achieves an effective balance between task performance and token efficiency, outperforming or matching the baselines across all tasks. On 10 out of 12 tasks, CARD shows better or comparable performance to policies trained with expert-designed rewards, and our method even surpasses the oracle on 3 tasks.

Paper Structure

This paper contains 44 sections, 1 equation, 7 figures, 21 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reward Design.
  4. Large Language Models for Reinforcement Learning.
  5. Problem Setup
  6. Method
  7. Reward Design
  8. Code Generation.
  9. Correctness Check.
  10. Reward Introspection
  11. Process Feedback
  12. Trajectory Feedback
  13. Trajectory Preference Evaluation and Preference Feedback
  14. Reward Improvement
  15. Practical Implementation
  16. ...and 29 more sections

Figures (7)

  • Figure 1: CARD includes a LLM-based Coder to generate reward function code and a Evaluator to evaluate the quality of the code. The Evaluator dynamically provides feedback to the Coder for reward function refinement.
  • Figure 2: Learning curves on six Meta-World tasks, measured by task success rate. The solid line represents the mean success rate, while the shaded regions correspond to the standard deviation, both calculated across five random seeds.
  • Figure 3: Learning curves on six ManiSkill2 tasks, measured by task success rate. The solid line represents the mean success rate, and the shaded areas denote the standard deviation, calculated across five random seeds.
  • Figure 4: Learning curves of CARD on Meta-World and ManiSkill2 tasks with different number of iterations. The solid line represents the mean success rate, and the shaded areas denote the standard deviation, calculated across five random seeds.
  • Figure 5: Learning curves of CARD on three Meta-World tasks with different types of feedback. The solid line represents the mean success rate, and the shaded areas denote the standard deviation, calculated across five random seeds.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Definition 4.1