Pixel to Elevation: Learning to Predict Elevation Maps at Long Range using Images for Autonomous Offroad Navigation

TL;DR

The paper tackles the challenge of predicting long-range terrain elevation for off-road navigation using only onboard monocular images. It introduces a transformer-based pipeline with three key innovations: cross-view attention in a multi-scale encoder, an orientation-aware positional encoding to embed vehicle pose, and a history-augmented map-view embedding to enforce temporal consistency. Empirical results on real-world off-road data show the approach outperforms state-of-the-art baselines in MAE and SDR at up to 100 m range while maintaining real-time performance (~41 FPS), with ablations confirming the value of OPE and HA. The work has practical implications for safer and more proactive planning in high-speed off-road robotics and motivates future integration of depth modalities and uncertainty estimation.

Abstract

Understanding terrain topology at long-range is crucial for the success of off-road robotic missions, especially when navigating at high-speeds. LiDAR sensors, which are currently heavily relied upon for geometric mapping, provide sparse measurements when mapping at greater distances. To address this challenge, we present a novel learning-based approach capable of predicting terrain elevation maps at long-range using only onboard egocentric images in real-time. Our proposed method is comprised of three main elements. First, a transformer-based encoder is introduced that learns cross-view associations between the egocentric views and prior bird-eye-view elevation map predictions. Second, an orientation-aware positional encoding is proposed to incorporate the 3D vehicle pose information over complex unstructured terrain with multi-view visual image features. Lastly, a history-augmented learn-able map embedding is proposed to achieve better temporal consistency between elevation map predictions to facilitate the downstream navigational tasks. We experimentally validate the applicability of our proposed approach for autonomous offroad robotic navigation in complex and unstructured terrain using real-world offroad driving data. Furthermore, the method is qualitatively and quantitatively compared against the current state-of-the-art methods. Extensive field experiments demonstrate that our method surpasses baseline models in accurately predicting terrain elevation while effectively capturing the overall terrain topology at long-ranges. Finally, ablation studies are conducted to highlight and understand the effect of key components of the proposed approach and validate their suitability to improve offroad robotic navigation capabilities.

Figures (6)

• Figure 1: Top row shows the robot navigating in an offroad natural environment during a field experiment conducted in Paso Robles, USA. The red box indicates the mounting position of the visual cameras used in this work, with the inset figure showing a zoomed-in view. Middle row shows the images taken by the left, front and right cameras during an instance of the experiment. Bottom row shows the elevation map output of the proposed method and compares it to the ground truth provided by the USGS. Using only visual camera images as input, our method is able to reliably predict the elevation map up to a distance of $100m$ to facilitate high-speed off-road robot navigation.
• Figure 2: An overview of our proposed architecture for terrain elevation map prediction. Front, left and right camera views are encoded at multiple scales using a shared backbone. The visual features are then transformed into the positional embedding using gravity-aligned vehicle orientation and camera matrix. Our history-augmented map-view query interacts with image features across from multiple views. Output of cross-attention module at each scale is upsampled to the desired dimensions and forwarded to the prediction head.
• Figure 3: Illustration of prediction area alignment over time for a history-augmented map-view embedding. $\mathcal{M}_t$ represents the elevation map prediction area at timestamp $t$ with respect to robot pose, $\mathcal{P}_t$. $\mathcal{O}_{t}$ represents the overlap area between predictions $\mathcal{M}_{t-1}$ and $\mathcal{M}_{t}$, represented with respect to pose $\mathcal{P}_t$.
• Figure 4: The figure shows examples of onboard camera images taken during field tests in two different environments, highlighting the variety of terrain and test conditions presented. Images from the Paso Robles site (top row) show grassy, hilly terrain with rapid changes in elevation. The Helendale site (bottom row) was a desert terrain containing scattered rocks and bushes.
• Figure 5: Demonstration of our results on the Paso Robles test set compared with SBEV, CVT, and Ground Truth (GT). The vehicle's direction is denoted by a red arrow, and the elevation at the vehicle's position always corresponds to zero.