Reward Guidance for Reinforcement Learning Tasks Based on Large Language Models: The LMGT Framework

TL;DR

LMGT presents a novel framework that injects prior knowledge from Large Language Models into reinforcement learning via reward shifting, enabling a data-efficient balance of exploration and exploitation. An LLM evaluator observes state–action pairs and outputs a reward shift $\delta r$, effectively reshaping the agent's rewards and accelerating learning while preserving standard RL workflows. Across Atari-like tasks, embodied robotics in Housekeep, and industrial recommendations with SlateQ, LMGT yields substantial improvements in sample efficiency and resource utilization, though it introduces LLM-inference overhead and requires careful prompt design. The work demonstrates strong empirical gains, offers ablations and real-world verifications, and outlines future theoretical and efficiency-oriented directions to broaden applicability. The approach holds practical potential for resource-constrained RL applications, where leveraging prior knowledge can dramatically reduce training costs without sacrificing performance.

Abstract

The inherent uncertainty in the environmental transition model of Reinforcement Learning (RL) necessitates a delicate balance between exploration and exploitation. This balance is crucial for optimizing computational resources to accurately estimate expected rewards for the agent. In scenarios with sparse rewards, such as robotic control systems, achieving this balance is particularly challenging. However, given that many environments possess extensive prior knowledge, learning from the ground up in such contexts may be redundant. To address this issue, we propose Language Model Guided reward Tuning (LMGT), a novel, sample-efficient framework. LMGT leverages the comprehensive prior knowledge embedded in Large Language Models (LLMs) and their proficiency in processing non-standard data forms, such as wiki tutorials. By utilizing LLM-guided reward shifts, LMGT adeptly balances exploration and exploitation, thereby guiding the agent's exploratory behavior and enhancing sample efficiency. We have rigorously evaluated LMGT across various RL tasks and evaluated it in the embodied robotic environment Housekeep. Our results demonstrate that LMGT consistently outperforms baseline methods. Furthermore, the findings suggest that our framework can substantially reduce the computational resources required during the RL training phase.

Figures (10)

• Figure 1: Schematic Representation of Diverse Approaches to Processing Environmental Information. It is evident that leveraging Visual Instruction Tuning in an end-to-end framework significantly enhances the capacity of LLMs to assimilate more pertinent information for informed decision-making, compared to the captioner-based approach.
• Figure 2: The structure of our LMGT framework. The LLM can observe the environment's state and the actions selected by the agent. It will evaluate the agent's behavior using prior knowledge, adjusting the final reward accordingly (via reward shifting). Thus, the agent’s stored experience inherently includes a component of prior knowledge.
• Figure 3: Performance comparison on challenging exploration Atari environments. The figures present reward curves demonstrating our LMGT method's effectiveness in (a) Pitfall and (b) Montezuma's Revenge - two environments characterized by extremely sparse rewards and complex exploration requirements. Our approach shows significant performance gains over baseline methods in these notoriously difficult benchmarks where conventional RL algorithms typically struggle to make progress.
• Figure 4: Results of experiments conducted across varying settings. It is important to note that all rewards in the Pendulum environment are negative. To enhance the visualization, each reward value shown in the graphs has been increased by an offset of 2000.
• Figure 5: Results from comparative experiments conducted within the Housekeep environment. Correct arrangement success rates on 4 object-receptacle task sets.