Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs

TL;DR

The paper tackles the difficulty of reward design in reinforcement learning for tasks with complex constraints by leveraging preference-based RL and large language models. It introduces LLM4PG, a framework that converts trajectories into natural language, uses LLMs to rank preferences under constraints, and learns a reward predictor that guides PPO optimization. Key contributions include the Language Interpreter for state abstraction, an LLM driven preference generator under complex constraints, and a reward modeling component trained from LLM feedback, demonstrated on MiniGrid tasks where it accelerates convergence and handles constraints more effectively than baselines. This work suggests a practical path to reducing domain knowledge requirements and expanding RL applicability in complex environments, with future directions including real-time rewards and multimodal LLMs for richer state descriptions.

Abstract

Reinforcement learning (RL) faces challenges in evaluating policy trajectories within intricate game tasks due to the difficulty in designing comprehensive and precise reward functions. This inherent difficulty curtails the broader application of RL within game environments characterized by diverse constraints. Preference-based reinforcement learning (PbRL) presents a pioneering framework that capitalizes on human preferences as pivotal reward signals, thereby circumventing the need for meticulous reward engineering. However, obtaining preference data from human experts is costly and inefficient, especially under conditions marked by complex constraints. To tackle this challenge, we propose a LLM-enabled automatic preference generation framework named LLM4PG , which harnesses the capabilities of large language models (LLMs) to abstract trajectories, rank preferences, and reconstruct reward functions to optimize conditioned policies. Experiments on tasks with complex language constraints demonstrated the effectiveness of our LLM-enabled reward functions, accelerating RL convergence and overcoming stagnation caused by slow or absent progress under original reward structures. This approach mitigates the reliance on specialized human knowledge and demonstrates the potential of LLMs to enhance RL's effectiveness in complex environments in the wild.

Figures (7)

• Figure 1: The overall framework of proposed LLM4PG.
• Figure 2: Success rate across the training process.
• Figure 3: The reward across the training process.
• Figure 4: Rewards and costs for each method.
• Figure 5: Rewards and costs under different $\lambda$.