Automatic occlusion removal from 3D maps for maritime situational awareness

TL;DR

A novel method for updating 3D geospatial models, specifically targeting occlusion removal in large-scale maritime environments, that leverages deep learning techniques to directly modify both the texture and geometry of 3D meshes without the need for costly reprocessing.

Abstract

We introduce a novel method for updating 3D geospatial models, specifically targeting occlusion removal in large-scale maritime environments. Traditional 3D reconstruction techniques often face problems with dynamic objects, like cars or vessels, that obscure the true environment, leading to inaccurate models or requiring extensive manual editing. Our approach leverages deep learning techniques, including instance segmentation and generative inpainting, to directly modify both the texture and geometry of 3D meshes without the need for costly reprocessing. By selectively targeting occluding objects and preserving static elements, the method enhances both geometric and visual accuracy. This approach not only preserves structural and textural details of map data but also maintains compatibility with current geospatial standards, ensuring robust performance across diverse datasets. The results demonstrate significant improvements in 3D model fidelity, making this method highly applicable for maritime situational awareness and the dynamic display of auxiliary information.

Figures (5)

• Figure 1: Overview of the proposed method for updating 3D geospatial models using mask-aware inpainting and geometric remeshing. First, an orthogonal view (bird's eye view, BEV) of the map is renderer in patches. Then, user-defined classes of objects are detected using instance segmentation. The resulting masks are used to control mask-aware inpainting of color and 3D position passes. The final step involves remeshing the geometry using the 3D position pass and reprojecting the inpainted color data onto the 3D model, resulting in an accurately adjusted representation of the environment.
• Figure 2: A qualitative comparison of our framework applied to a 3D geospatial map inpainted using LaMa suvorov2021resolution. The left side depicts the original 3D scene, while the middle illustrates the results after inpainting and remeshing. Occluded areas and dynamic features have been seamlessly reconstructed and updated. The highlighted regions on the far right demonstrate the fidelity of our technique, which effectively preserves structural details and ensures consistent and accurate updates to the geospatial data.
• Figure 3: Heatmap of the normalized distance between source elevation and inpainted elevation for an example patch. The heatmap was overlaid on the source image using the mask generated by the instance segmentation described in this section. False-colored areas indicate regions where the model correctly identified and updated occlusions, demonstrating the accuracy of our mask-aware inpainting approach.
• Figure 4: Qualitative comparison of inpainting methods on color and position maps: (a) Source BEV, (b, f) CoModGANzhao2021comodgan, (c, g) MATmat2022, (d, h) LaMasuvorov2021resolution, (e) merged and dilated mask from instance segmentation. Note how CoModGAN performs comparable to LaMa for position data, while MAT fails to remove the ship. For color data LaMa outperforms both MAT and CoModGAN for color data leaving only a few shadow artifacts. For quantitative analysis, refer to Table 1.
• Figure 5: Wireframe comparison: (Left) Original 3D mesh structure, (Right) Remeshed structure after projection and applying a distance-based merge. While there are remnants of the original mesh in the inpainted parts, the topology is consistent.