Spectral Signature Mapping from RGB Imagery for Terrain-Aware Navigation

TL;DR

This work tackles the limitation of RGB-only perception in terrain interaction by predicting spectral signatures from RGB images through RS-Net, enabling terrain classification and friction estimation without costly spectral hardware. The model integrates a DenseNet-based feature extractor with a spectral projection head and a lightweight task-specific head, trained end-to-end with a mix of spectral reconstruction and task losses. RS-Net demonstrates improved spectral reconstruction over baselines, enables friction-aware control in a quadruped, and guides terrain-aware planning in a wheeled robot, achieving real-time performance (~5 Hz). The approach offers a practical, low-cost pathway to incorporate material properties into autonomous navigation across outdoor environments.

Abstract

Successful navigation in outdoor environments requires accurate prediction of the physical interactions between the robot and the terrain. Many prior methods rely on geometric or semantic labels to classify traversable surfaces. However, such labels cannot distinguish visually similar surfaces that differ in material properties. Spectral sensors enable inference of material composition from surface reflectance measured across multiple wavelength bands. Although spectral sensing is gaining traction in robotics, widespread deployment remains constrained by the need for custom hardware integration, high sensor costs, and compute-intensive processing pipelines. In this paper, we present the RGB Image to Spectral Signature Neural Network (RS-Net), a deep neural network designed to bridge the gap between the accessibility of RGB sensing and the rich material information provided by spectral data. RS-Net predicts spectral signatures from RGB patches, which we map to terrain labels and friction coefficients. The resulting terrain classifications are integrated into a sampling-based motion planner for a wheeled robot operating in outdoor environments. Likewise, the friction estimates are incorporated into a contact-force-based MPC for a quadruped robot navigating slippery surfaces. Overall, our framework learns the task-relevant physical properties offline during training and thereafter relies solely on RGB sensing at run time.

Figures (11)

• Figure 1: Our approach takes RGB image patches (Left) as input and predicts their spectral signatures (Middle). These predicted spectra are then mapped to terrain classes by tuning the final fully connected layer (Right). The resulting terrain predictions inform navigation. The robot selects a path along the asphalt road, which is easier to traverse than the surrounding dense grass.
• Figure 2: Architecture to predict spectral signature $x_s'$ from RGB image $x$.
• Figure 3: Overview of our motion planning pipeline. Segment Anything (SAM) extracts image patches from the RGB input, which are passed through RS-Net followed by a Fully Connected Network to predict per-patch friction estimates or terrain labels. These predictions inform friction constraints or terrain-aware cost maps for downstream motion planning algorithms.
• Figure 4: Top: RGB patch of gravel. Bottom: RGB patch of asphalt. RS-Net predicts distinct spectral signatures for these visually similar surfaces.
• Figure 5: Representative surface textures used in simulation. From left to right: asphalt, brick, and ice.