Semantic UV mapping to improve texture inpainting for indoor scenes

TL;DR

A new UV mapping pre-processing step which leverages semantic information of indoor scenes to more accurately match the UV islands with the 3D representation of distinct structural elements like walls and floors is achieved.

Abstract

This work aims to improve texture inpainting after clutter removal in scanned indoor meshes. This is achieved with a new UV mapping pre-processing step which leverages semantic information of indoor scenes to more accurately match the UV islands with the 3D representation of distinct structural elements like walls and floors. Semantic UV Mapping enriches classic UV unwrapping algorithms by not only relying on geometric features but also visual features originating from the present texture. The segmentation improves the UV mapping and simultaneously simplifies the 3D geometric reconstruction of the scene after the removal of loose objects. Each segmented element can be reconstructed separately using the boundary conditions of the adjacent elements. Because this is performed as a pre-processing step, other specialized methods for geometric and texture reconstruction can be used in the future to improve the results even further.

Figures (7)

• Figure 1: Overview of the proposed pipeline, starting with a furnished mesh (left), featuring the parallel scene segmentation and geometric reconstruction (top), and the semantic UV mapping and texture reconstruction (bottom) to result in an empty room mesh (right).
• Figure 2: The two-plane-intersection case (left) and the three-plane-intersection case (right)
• Figure 3: A scene from the ScanNet++ dataset (left) and Matterport3D dataset (right)
• Figure 4: The matterport room(left) and the segmented labels (right)
• Figure 5: The object-removed room (left) and the reconstructed room (right)