STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft

TL;DR

This work tackles the challenge of building embodied agents powered by large language models for open-world task execution, using Minecraft as a controllable testbed to balance cost and reliability. It introduces the STEVE Series, starting from a vanilla LLM augmented with a vision encoder and a STEVE-21K action dataset, then adding a Critic, memory, and a hierarchical multi-agent system, followed by hierarchical knowledge distillation to a single model. The approach yields substantial efficiency gains of $2.5\times$ to $7.3\times$ and demonstrates strong performance across basic skills, navigation, and creation on MineDojo-based benchmarks. The results indicate significant potential for scalable, multimodal, and cooperative embodied agents in real-world settings, supported by a rich dataset and modular architecture that can be extended to complex environments.

Abstract

Building an embodied agent system with a large language model (LLM) as its core is a promising direction. Due to the significant costs and uncontrollable factors associated with deploying and training such agents in the real world, we have decided to begin our exploration within the Minecraft environment. Our STEVE Series agents can complete basic tasks in a virtual environment and more challenging tasks such as navigation and even creative tasks, with an efficiency far exceeding previous state-of-the-art methods by a factor of $2.5\times$ to $7.3\times$. We begin our exploration with a vanilla large language model, augmenting it with a vision encoder and an action codebase trained on our collected high-quality dataset STEVE-21K. Subsequently, we enhanced it with a Critic and memory to transform it into a complex system. Finally, we constructed a hierarchical multi-agent system. Our recent work explored how to prune the agent system through knowledge distillation. In the future, we will explore more potential applications of STEVE agents in the real world.

Figures (1)

• Figure 1: STEVE Series overview.