Leveraging LLMs for reward function design in reinforcement learning control tasks

TL;DR

LEARN-Opt presents an autonomous, LLM-driven framework for reward function design in RL control tasks, removing dependence on predefined metrics or environment code. It jointly introduces a problem formulation where the optimal reward function $r^*$ maximizes a fitness $F$ derived entirely from system descriptions, and implements a closed-loop workflow with generation, execution, and evaluation modules. Through an ensemble of analyzers and a multi-run strategy, LEARN-Opt achieves comparable or superior performance to EUREKA across several benchmarks while demonstrating the ability to use low-cost LLMs effectively. The work highlights the high-variance nature of automated reward design and suggests practical guidelines for analyzer configuration and model choice, underscoring its potential to generalize reward design with reduced engineering overhead.

Abstract

The challenge of designing effective reward functions in reinforcement learning (RL) represents a significant bottleneck, often requiring extensive human expertise and being time-consuming. Previous work and recent advancements in large language models (LLMs) have demonstrated their potential for automating the generation of reward functions. However, existing methodologies often require preliminary evaluation metrics, human-engineered feedback for the refinement process, or the use of environmental source code as context. To address these limitations, this paper introduces LEARN-Opt (LLM-based Evaluator and Analyzer for Reward functioN Optimization). This LLM-based, fully autonomous, and model-agnostic framework eliminates the need for preliminary metrics and environmental source code as context to generate, execute, and evaluate reward function candidates from textual descriptions of systems and task objectives. LEARN-Opt's main contribution lies in its ability to autonomously derive performance metrics directly from the system description and the task objective, enabling unsupervised evaluation and selection of reward functions. Our experiments indicate that LEARN-Opt achieves performance comparable to or better to that of state-of-the-art methods, such as EUREKA, while requiring less prior knowledge. We find that automated reward design is a high-variance problem, where the average-case candidate fails, requiring a multi-run approach to find the best candidates. Finally, we show that LEARN-Opt can unlock the potential of low-cost LLMs to find high-performing candidates that are comparable to, or even better than, those of larger models. This demonstrated performance affirms its potential to generate high-quality reward functions without requiring any preliminary human-defined metrics, thereby reducing engineering overhead and enhancing generalizability.

Figures (9)

• Figure 1: Main related works classification: By generation requirements (left). By evaluation feedback (right).
• Figure 2: Overall workflow of the LEARN-Opt framework: The process begins with the system description and the reward function specification. The generator uses this information to generate candidate reward functions, which are sent to the execution module. After the reward function candidates are tested and their results are collected, these results are passed to the evaluation module, which determines the best reward function candidates in each loop of the optimization process.
• Figure 3: Generator module. The mapping agent maps the system description's states and actions and returns this information to the ZS/FS agent to generate valid candidates.
• Figure 4: Execution module.
• Figure 5: Evaluation module. The coder agent is used for coding purposes, while the planner agent is used for a chain-of-thought procedure to generate an evaluation and selection procedure. Its architecture allows the creation of different analyzer sub-modules so that a council can determine the best candidate.