HEADER: Hierarchical Robot Exploration via Attention-Based Deep Reinforcement Learning with Expert-Guided Reward

TL;DR

HEADER tackles large-scale autonomous robot exploration by learning a scalable, hierarchical planner. It combines a community-based global graph constructed with the Leiden algorithm, an attention-based local planner, and a parameter-free privileged expert reward to reduce reward-bias. The method enables joint global-local decisions through guideposts, improving scalability and exploration efficiency in both simulation and hardware, including a $300m\times230m$ campus and over $60{,}000\mathrm{m}^3$ of space. Across benchmarks, HEADER outperforms state-of-the-art baselines by striking a favorable balance between global planning and local adaptivity and demonstrates robust sim-to-real transferability.

Abstract

This work pushes the boundaries of learning-based methods in autonomous robot exploration in terms of environmental scale and exploration efficiency. We present HEADER, an attention-based reinforcement learning approach with hierarchical graphs for efficient exploration in large-scale environments. HEADER follows existing conventional methods to construct hierarchical representations for the robot belief/map, but further designs a novel community-based algorithm to construct and update a global graph, which remains fully incremental, shape-adaptive, and operates with linear complexity. Building upon attention-based networks, our planner finely reasons about the nearby belief within the local range while coarsely leveraging distant information at the global scale, enabling next-best-viewpoint decisions that consider multi-scale spatial dependencies. Beyond novel map representation, we introduce a parameter-free privileged reward that significantly improves model performance and produces near-optimal exploration behaviors, by avoiding training objective bias caused by handcrafted reward shaping. In simulated challenging, large-scale exploration scenarios, HEADER demonstrates better scalability than most existing learning and non-learning methods, while achieving a significant improvement in exploration efficiency (up to 20%) over state-of-the-art baselines. We also deploy HEADER on hardware and validate it in complex, large-scale real-life scenarios, including a 300m*230m campus environment.

Figures (11)

• Figure 1: Example hierarchical graph constructed by HEADER during its autonomous exploration of our campus. While dense local nodes keep expanding to cover all traversable areas, our planner adaptively and incrementally partitions these local nodes (bounded by the blue box as the local range) into global nodes to generate the global reference path. Based on the local graphs and the global path, our trained model selects a neighboring node as the next waypoint (denoted as the purple ball) to visit.
• Figure 2: Overall working flow of HEADER during deployment. We first classify the sensor scan into free and obstacle point clouds, which are used to build the current map of the environment. Then, we construct the community-based hierarchical graph to generate the graph observation as the input of the trained neural network. After that, our attention-based decision model outputs the next waypoint for the robot to follow and get new measurements.
• Figure 3: Example community-based hierarchical graph constructed by HEADER. The purple dot indicates the robot's current position, and the green square the local graph region. Small dots denote local nodes, and blue squares global nodes. The blue paths denote the global edges, and the red path the planned global path. In Fig. \ref{['fig:global_graph']}, the colors of local nodes represent their community memberships, while in Fig. \ref{['fig:node_utility']}, they reflect node utilities. Fig. \ref{['fig:local_guidepost']} and \ref{['fig:global_guidepost']} highlight the selected local and global guideposts, respectively. Local guideposts denote paths to all utility nodes, and global guideposts paths to the next global node.
• Figure 4: Attention-based decision network in HEADER. It takes the local graph, utility, and guideposts as input and outputs a probability distribution over neighboring nodes of the current robot position as its policy.
• Figure 5: Illustration of the privileged expert path. The white area denotes the explored free space, while the light gray region indicates unexplored free space. The red trajectory represents the privileged expert path planned using ground-truth information. The yellow dot marks the expert path's first point as the expert waypoint. The light purple dot indicates the robot's current position, and the dark purple dots represent the candidate waypoints. Our privileged reward is based on the distance between the selected waypoint and the expert waypoint.