REvolve: Reward Evolution with Large Language Models using Human Feedback

TL;DR

REvolve reframes reward design for reinforcement learning as an evolutionary search guided by human feedback, using GPT-4 as a reward designer and genetic programming with an island model to evolve executable Python reward functions. By treating human preferences as non-differentiable fitness signals and deploying LLMs as intelligent genetic operators, REvolve avoids training separate reward models and yields interpretable, task-specific rewards. Across autonomous driving, humanoid locomotion, and dexterous manipulation, REvolve outperforms greedy LLM-based methods like Eureka and surpasses baselines, approaching human expert performance in driving while maintaining diverse reward candidates. The approach highlights a scalable path to human-aligned RL in complex, tacitly defined tasks, though it entails high compute and depends on proprietary LLMs, pointing to future work in open models and real-world transfer.

Abstract

Designing effective reward functions is crucial to training reinforcement learning (RL) algorithms. However, this design is non-trivial, even for domain experts, due to the subjective nature of certain tasks that are hard to quantify explicitly. In recent works, large language models (LLMs) have been used for reward generation from natural language task descriptions, leveraging their extensive instruction tuning and commonsense understanding of human behavior. In this work, we hypothesize that LLMs, guided by human feedback, can be used to formulate reward functions that reflect human implicit knowledge. We study this in three challenging settings -- autonomous driving, humanoid locomotion, and dexterous manipulation -- wherein notions of ``good" behavior are tacit and hard to quantify. To this end, we introduce REvolve, a truly evolutionary framework that uses LLMs for reward design in RL. REvolve generates and refines reward functions by utilizing human feedback to guide the evolution process, effectively translating implicit human knowledge into explicit reward functions for training (deep) RL agents. Experimentally, we demonstrate that agents trained on REvolve-designed rewards outperform other state-of-the-art baselines.

Figures (13)

• Figure 1: An overview of REvolve. Given the task (here, autonomous driving) and abstracted environment variables, a reward designer $G$ (LLM) outputs a population of reward functions $R \in \mathcal{R}$, each used to train a policy $\pi(R)$ in simulation. Then, we collect human preferences and natural language (NL) feedback on pairs of policy rollouts $\theta \sim \Theta_{\pi(R)}$ through a user feedback interface. Policy (and thus, corresponding reward function) fitness $\sigma$ is calculated, and the fittest individuals, along with their NL feedback $\lambda$, are refined by $G$. In addition, the process leverages genetic programming for evolution.
• Figure 2: Illustration of how GPT-4 applies mutation and crossover to reward functions. Mutation (left): shows the modification of the "smoothness reward" component. A red '-' sign indicates the line removed from the parent reward function, while a green '+' sign indicates the line added to the new reward function. Crossover (right): demonstrates how parent reward functions are combined to create a child reward function, incorporating the most effective reward components from each parent.
• Figure 3: We plot the fitness score -- over 2 random seeds -- of the best-performing policy per generation for autonomous driving (left), humanoid locomotion (middle), and adroit hand manipulation (right), cf. Table \ref{['tab:rl_tasks_summary']}. Each plot also contains the fitness score obtained using an expert-designed reward function. The fitness score for the human driving in the autonomous driving setting (left) was obtained by observing the driving behavior of an expert human conducting driving sessions in a combined setting of AirSim and the high-fidelity Logitech driving simulator. We treat this as the upper limit of performance. We give the formal definitions of the fitness functions for each of the three environments in Equations \ref{['eq:fitness_ad']}, \ref{['eq:fitness_hl']}, and \ref{['eq:fitness_hm']}, respectively (in Appendix \ref{['appendix: eureka automatic feedback']}).
• Figure 4: Comparison of REvolve's performance on Autonomous Driving using mutation only, crossover only, and both combined, over 7 generations.
• Figure 5: AirSim simulator environment used for our autonomous driving experiments. The egocentric camera view displayed on the bottom left serves as the observations for the autonomous driving agent.