Inpainting the Red Planet: Diffusion Models for the Reconstruction of Martian Environments in Virtual Reality

TL;DR

We address the reconstruction of missing Martian heightmaps for VR-based visualization. We train an unconditional diffusion model on a multi-scale HiRISE-derived dataset to fill gaps without relying on conditioning data, producing outputs at $128 \times 128$ resolution. The method is compared against Inverse Distance Weighting, kriging, and Navier-Stokes inpainting, achieving up to 4–15% improvements in RMSE and 29–81% gains in perceptual similarity (LPIPS). The results demonstrate geometrically coherent, VR-ready terrain reconstructions that enable more reliable mission planning and immersive simulations.

Abstract

Space exploration increasingly relies on Virtual Reality for several tasks, such as mission planning, multidisciplinary scientific analysis, and astronaut training. A key factor for the reliability of the simulations is having accurate 3D representations of planetary terrains. Extraterrestrial heightmaps derived from satellite imagery often contain missing values due to acquisition and transmission constraints. Mars is among the most studied planets beyond Earth, and its extensive terrain datasets make the Martian surface reconstruction a valuable task, although many areas remain unmapped. Deep learning algorithms can support void-filling tasks; however, whereas Earth's comprehensive datasets enables the use of conditional methods, such approaches cannot be applied to Mars. Current approaches rely on simpler interpolation techniques which, however, often fail to preserve geometric coherence. In this work, we propose a method for reconstructing the surface of Mars based on an unconditional diffusion model. Training was conducted on an augmented dataset of 12000 Martian heightmaps derived from NASA's HiRISE survey. A non-homogeneous rescaling strategy captures terrain features across multiple scales before resizing to a fixed 128x128 model resolution. We compared our method against established void-filling and inpainting techniques, including Inverse Distance Weighting, kriging, and Navier-Stokes algorithm, on an evaluation set of 1000 samples. Results show that our approach consistently outperforms these methods in terms of reconstruction accuracy (4-15% on RMSE) and perceptual similarity (29-81% on LPIPS) with the original data.

Figures (8)

• Figure 1: The generative capabilities of diffusion models provide realistic restorations of degraded acquisitions, producing structurally consistent representations suitable for virtual environments. (Left) A degraded Martian heightmap depicting impact craters, with regions composed of missing values. (Right) The same heightmap after reconstruction, using our diffusion-based approach; the inpainted regions are explicitly highlighted.
• Figure 2: RGB representation of Martian surfaces, obtained from the processing of HiRISE stereo pairs. It may occur that missing data are interpolated during processing (the squared patterns within the crater in the left image) or entirely unavailable (the black line in the right image).
• Figure 3: Evaluation of the trained unconditional DDPM after 100 epochs of training with Martian terrains.
• Figure 4: Input (left) and output (right) of our approach to Martian surface reconstruction, represented as 2D rasters.
• Figure 5: Example mask shapes that were used for evaluation. The masks are artificially generated to simulate common patterns that occur in data losses.