Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft

TL;DR

The paper tackles sparse rewards in Minecraft RL by introducing Auto MC-Reward, an LLM-driven framework that automatically designs, verifies, and refines dense reward functions through three components: Reward Designer, Reward Critic, and Trajectory Analyzer. It outputs executable, explainable reward functions with multi-step memory and chain-of-thought prompts, which are tested and refined via a structured feedback loop. Empirical results in MineDojo show that Auto MC-Reward improves learning efficiency and task success over sparse rewards and several dense-reward baselines, including long-horizon diamond mining tasks. The work demonstrates a scalable, interpretable approach to automated dense reward design for open-world, sparse-reward reinforcement learning.

Abstract

Many reinforcement learning environments (e.g., Minecraft) provide only sparse rewards that indicate task completion or failure with binary values. The challenge in exploration efficiency in such environments makes it difficult for reinforcement-learning-based agents to learn complex tasks. To address this, this paper introduces an advanced learning system, named Auto MC-Reward, that leverages Large Language Models (LLMs) to automatically design dense reward functions, thereby enhancing the learning efficiency. Auto MC-Reward consists of three important components: Reward Designer, Reward Critic, and Trajectory Analyzer. Given the environment information and task descriptions, the Reward Designer first design the reward function by coding an executable Python function with predefined observation inputs. Then, our Reward Critic will be responsible for verifying the code, checking whether the code is self-consistent and free of syntax and semantic errors. Further, the Trajectory Analyzer summarizes possible failure causes and provides refinement suggestions according to collected trajectories. In the next round, Reward Designer will further refine and iterate the dense reward function based on feedback. Experiments demonstrate a significant improvement in the success rate and learning efficiency of our agents in complex tasks in Minecraft, such as obtaining diamond with the efficient ability to avoid lava, and efficiently explore trees and animals that are sparse in the plains biome.

Figures (8)

• Figure 1: Overview of our Auto MC-Reward. Auto MC-Reward consists of three key LLM-based components: Reward Designer, Reward Critic, and Trajectory Analyzer. A suitable dense reward function is iterated through the continuous interaction between the agent and the environment for reinforcement learning training of specific tasks, so that the model can better complete the task. An example of exploring diamond ore is shown in the figure: i) Trajectory Analyzer finds that the agent dies from lava in the failed trajectory, and then gives suggestion for punishment when encountering lava; ii) Reward Designer adopts the suggestion and updates the reward function; iii) The revised reward function passes the review of Reward Critic, and finally the agent avoids the lava by turning left.
• Figure 2: Example of updating the reward function. Trajectory Analyzer provides analysis for three scenarios at different steps, and then Reward Designer update the reward function based on the suggestions. We only display part of the trajectory data for brevity. Step 1: rewrite the code of encouraging exploration to avoid going back and forth. Step 2: add lava penalty to avoid falling into lava. Step 3: add pitch constraint to avoid constantly looking up to avoid lava.
• Figure 3: The trajectories of the new behaviors. (a) Avoid lava when exploring for diamond ore . (b) Attack cow in plains.
• Figure 4: The tech tree of obtaining diamond. The green squares are tasks to be optimized with Auto MC-Reward, i.e. obtaining log , cobblestone , iron ore and diamond .
• Figure 5: Cumulative success rates for 4 key items of obtaining diamond on forest and plains . In terms of diamond, the performance comparison between imitation learning and Auto MC-Reward in two biomes are: 28.8% vs. 36.5%, and 0% vs. 28.1%.