SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competition

TL;DR

SEIHAI tackles the ObtainDiamond problem under sparse rewards by a fully automatic hierarchical framework that divides the task into five subtasks aligned with an item dependency, each handled by a dedicated subagent trained with RL or imitation learning. A learning-based scheduler selects the appropriate agent in state, enabling robust composition from limited demonstrations and domain-agnostic action spaces. The approach achieves top results in the NeurIPS-2020 MineRL competition, highlighting the value of priors and modular, data-efficient learning for complex, long-horizon tasks. The formal MDP is defined as $\langle S,A,T,R,\gamma\rangle$ with return $G=\sum_{t=0}^H \gamma^t r_t$, illustrating the balance between structure and learning in sample-efficient AI for real-world-like environments.

Abstract

The MineRL competition is designed for the development of reinforcement learning and imitation learning algorithms that can efficiently leverage human demonstrations to drastically reduce the number of environment interactions needed to solve the complex \emph{ObtainDiamond} task with sparse rewards. To address the challenge, in this paper, we present \textbf{SEIHAI}, a \textbf{S}ample-\textbf{e}ff\textbf{i}cient \textbf{H}ierarchical \textbf{AI}, that fully takes advantage of the human demonstrations and the task structure. Specifically, we split the task into several sequentially dependent subtasks, and train a suitable agent for each subtask using reinforcement learning and imitation learning. We further design a scheduler to select different agents for different subtasks automatically. SEIHAI takes the first place in the preliminary and final of the NeurIPS-2020 MineRL competition.

Figures (4)

• Figure 1: The background about the MineRL competition: the grey rectangles represent different items, e.g, Log, Plank, Crafting_Table, Stick, Wooden_Pickaxe, and so on; the arrows demonstrate the typical item hierarchy for obtaining a Diamond; the red texts denote the reward when the agent mines or crafts an item for the first time. We split the ObtainDiamond task into five subtasks according to the natural hierarchy among items, and train a suitable agent for each subtask.
• Figure 2: The state and action spaces in the ObtainDiamond task. The obfuscated states and actions are more complex since we do not know their semantic meanings.
• Figure 3: The proposed architecture of our method.
• Figure 4: The Division Points of an Episode.