Hybrid Belief Reinforcement Learning for Efficient Coordinated Spatial Exploration

TL;DR

A hybrid belief-reinforcement learning (HBRL) framework that enables coordinated coverage through shared belief state, permitting cooperative sensing in high-uncertainty regions while discouraging redundant coverage in well-explored areas is presented.

Abstract

Coordinating multiple autonomous agents to explore and serve spatially heterogeneous demand requires jointly learning unknown spatial patterns and planning trajectories that maximize task performance. Pure model-based approaches provide structured uncertainty estimates but lack adaptive policy learning, while deep reinforcement learning often suffers from poor sample efficiency when spatial priors are absent. This paper presents a hybrid belief-reinforcement learning (HBRL) framework to address this gap. In the first phase, agents construct spatial beliefs using a Log-Gaussian Cox Process (LGCP) and execute information-driven trajectories guided by a Pathwise Mutual Information (PathMI) planner with multi-step lookahead. In the second phase, trajectory control is transferred to a Soft Actor-Critic (SAC) agent, warm-started through dual-channel knowledge transfer: belief state initialization supplies spatial uncertainty, and replay buffer seeding provides demonstration trajectories generated during LGCP exploration. A variance-normalized overlap penalty enables coordinated coverage through shared belief state, permitting cooperative sensing in high-uncertainty regions while discouraging redundant coverage in well-explored areas. The framework is evaluated on a multi-UAV wireless service provisioning task. Results show 10.8% higher cumulative reward and 38% faster convergence over baselines, with ablation studies confirming that dual-channel transfer outperforms either channel alone.

Figures (17)

• Figure 1: Multiple mobile agents spatial exploration over a discretized operational area with unknown demand, modeled via a spatial belief process. The illustrated scenario depicts UAVs as the instantiated agents, with further details provided in Section 4.
• Figure 2: Overview of the proposed HBRL framework. Phase 1 employs LGCP-based belief inference and PathMI planning to guide information-driven exploration. Phase 2 performs policy optimization using Soft Actor-Critic (SAC), warm-started via dual-channel knowledge transfer: (i) belief state initialization and (ii) replay buffer seeding with LGCP-generated trajectories.
• Figure 3: Two-phase training pipeline: LGCP--PathMI exploration (left) and warm-started SAC training (right).
• Figure 4: Reward comparison between Pure LGCP, Pure RL, Behavior Cloning and HBRL frameworks.
• Figure 5: Posterior Variance comparison between Pure LGCP, Pure RL and HBRL. Lower values indicate higher confidence in the inferred demand field.