Heterogeneous Multi-Expert Reinforcement Learning for Long-Horizon Multi-Goal Tasks in Autonomous Forklifts

TL;DR

The paper tackles the challenge of long-horizon, multi-goal autonomous forklift manipulation in unstructured warehouses by introducing HMER, a heterogeneous multi-expert reinforcement learning framework. HMER decouples macro navigation from micro manipulation using modality-specific experts (Navigation, Picking, Placing) orchestrated by a Hierarchical Semantic Task Planner, addressing optimization interference and error propagation. A Hybrid Imitation-Reinforcement Training strategy initializes policies via Behavioral Cloning and then refines them with Residual PPO, achieving sub-centimeter placement accuracy (≈1.5 cm) and high task success (≈94.2%) in Gazebo simulations. Results show superior performance and robustness over end-to-end and sequential baselines, with implications for improved industrial throughput and a clear path toward Sim-to-Real transfer and multi-agent collaboration.

Abstract

Autonomous mobile manipulation in unstructured warehouses requires a balance between efficient large-scale navigation and high-precision object interaction. Traditional end-to-end learning approaches often struggle to handle the conflicting demands of these distinct phases. Navigation relies on robust decision-making over large spaces, while manipulation needs high sensitivity to fine local details. Forcing a single network to learn these different objectives simultaneously often causes optimization interference, where improving one task degrades the other. To address these limitations, we propose a Heterogeneous Multi-Expert Reinforcement Learning (HMER) framework tailored for autonomous forklifts. HMER decomposes long-horizon tasks into specialized sub-policies controlled by a Semantic Task Planner. This structure separates macro-level navigation from micro-level manipulation, allowing each expert to focus on its specific action space without interference. The planner coordinates the sequential execution of these experts, bridging the gap between task planning and continuous control. Furthermore, to solve the problem of sparse exploration, we introduce a Hybrid Imitation-Reinforcement Training Strategy. This method uses expert demonstrations to initialize the policy and Reinforcement Learning for fine-tuning. Experiments in Gazebo simulations show that HMER significantly outperforms sequential and end-to-end baselines. Our method achieves a task success rate of 94.2\% (compared to 62.5\% for baselines), reduces operation time by 21.4\%, and maintains placement error within 1.5 cm, validating its efficacy for precise material handling.

Figures (6)

• Figure 1: Illustration of the Long-Horizon Material Handling Task. The operational lifecycle is decomposed into four sequential phases: (1) Departure from the docking station $\mathcal{P}_{start}$; (2) Search & Pick at the randomized cargo location $\mathcal{P}_{obj}$; (3) Transport through dynamic obstacles; and (4) Precision Placement at the target slot $\mathcal{P}_{goal}$. The heterogeneous sensory inputs (LiDAR for navigation, RGB for manipulation) act as triggers for the state transitions.
• Figure 2: The HMER Framework Architecture. The high-level Semantic Task Planner observes discrete semantic states to orchestrate active low-level experts. Each expert is structurally decoupled, utilizing specialized encoders for distinct modalities (sparse geometric LiDAR vs. dense semantic RGB) to eliminate optimization interference.
• Figure 3: Semantic Task Planner Logic. The finite state machine governs transitions between operational phases based on semantic predicates, acting as a manifold constrainer for the learning agents.
• Figure 4: Hybrid Imitation-Reinforcement Training Strategy. Phase 1 utilizes Behavioral Cloning for stable manifold initialization. Phase 2 employs PPO for end-to-end residual refinement, enabling the experts to learn complex contact dynamics.
• Figure 5: Training Dynamics Analysis. Task success rate vs. environment interaction steps. HMER (Ours) achieves faster convergence and higher asymptotic performance compared to HRL from scratch, while monolithic baselines fail to learn.