Fast Navigation Through Occluded Spaces via Language-Conditioned Map Prediction

TL;DR

This work tackles safe, fast navigation under occlusion by introducing PaceForecaster, which conditions Level-2 map forecasts and a subgoal on language instructions $I_t$ and the current local map $M_t$. The system uses a FiLM-conditioned U-Net to predict $F_t$ and a second head to predict a Gaussian heatmap for $G^{(2)}_t$, with joint training and integration into a Log-MPPI controller. Across Visdiff-based polygon environments, end-to-end and simulation experiments show substantial improvements over Level-1-only baselines, including up to 46% gains with ground-truth forecasts and a 36% end-to-end improvement with predicted forecasts, validating the value of language-conditioned look-ahead for occluded navigation. The approach also demonstrates a practical NL-to-symbolic instruction conversion pipeline, enabling real-world deployment with co-pilot-style guidance.

Abstract

In cluttered environments, motion planners often face a trade-off between safety and speed due to uncertainty caused by occlusions and limited sensor range. In this work, we investigate whether co-pilot instructions can help robots plan more decisively while remaining safe. We introduce PaceForecaster, as an approach that incorporates such co-pilot instructions into local planners. PaceForecaster takes the robot's local sensor footprint (Level-1) and the provided co-pilot instructions as input and predicts (i) a forecasted map with all regions visible from Level-1 (Level-2) and (ii) an instruction-conditioned subgoal within Level-2. The subgoal provides the planner with explicit guidance to exploit the forecasted environment in a goal-directed manner. We integrate PaceForecaster with a Log-MPPI controller and demonstrate that using language-conditioned forecasts and goals improves navigation performance by 36% over a local-map-only baseline while in polygonal environments.

Figures (10)

• Figure 1: Co-Driver Setting: (a) Assume the robot approaches a junction with occlusions in both left and right with no information on the direction and the sharpness of the turn. (b) The co-driver gives the instruction of "Sharp Right Turn". We seek to answer the question - (c) Can we generate the turn and the goal (shown in green tick mark) based on the language instruction of the co-driver so that the car can generate the path shown in red.
• Figure 2: Level-1 vs Level-2 Maps: Level-1 shows the local LiDAR map in the robot frame (robot shown in red). Level-2 is the region that would become visible if a LiDAR scan was taken from all of the frontier endpoints of the Level-1 map (yellow). The bottom figure shows the final occupancy maps of Level-1 and Level-2 where violet represents unknown, blue represents free space and yellow represents obstacle regions.
• Figure 3: PaceForecaster architecture: There are two main modules: U-Net Level-2 Prediction and Subgoal Prediction. U-Net Level-2 Prediction: First, the Level-1 local map in the robot frame is provided as input to a U-Net, which generates the Level-2 predicted map conditioned on the instruction encoding. Subgoal Prediction: Given the predicted Level-2 map as input, the subgoal head outputs a heatmap of the subgoal location, also conditioned on the instruction encoding. Conditioning is applied via FiLM perez2017filmvisualreasoninggeneral to the spatial CNN features in both the map and subgoal prediction networks. During training, the Level-2 map and subgoal prediction models are optimized jointly. In the occupancy grid representation of the map, violet denotes unknown regions, yellow denotes obstacles, and blue denotes free space.
• Figure 4: GT Level-2 vs Local Map Navigation Qualitative Results: At high speed ($3\,\mathrm{m/s}$) in narrow passages, access to ground-truth Level-2 (GT-L2) enables planning ahead and successful turning. Zoomed-in views show that the GT-L2 trajectory (green) approaches the corner with an appropriate entry angle, whereas relying only on the local map (red) fails to complete the turn.
• Figure 5: Unimodal Prediction Qualitative: This result shows the model’s prediction of the Level-2 environment and goal, given that only one opening is visible in the environment and the language instruction specifies going straight into the L1 turn. The blue arrow shows the robot, while the red cross marks the goals. Violet refers to the unknown area in the occupancy grid, yellow refers to the obstacle area, and blue refers to the free space.