Interacted Planes Reveal 3D Line Mapping

TL;DR

LiP-Map tackles the problem of reconstructing structured 3D line maps by coupling learnable 3D planar primitives with 3D line edges. It jointly optimizes planes and plane-edge lines using depth/normal supervision and 2D line detections, avoiding fragile 2D line matching by mapping detections to plane edges via an attraction-field guided assignment. The approach demonstrates strong gains in accuracy and completeness across ScanNetV2, ScanNet++, and Hypersim, and robust line-assisted visual localization on 7Scenes, while maintaining practical efficiency (3–5 minutes per scene). By explicitly modeling planar topology and its interaction with lines, LiP-Map provides a principled, scalable route to structured indoor scene reconstruction and compact geometric representations with broad applicability across robotics and AR.

Abstract

3D line mapping from multi-view RGB images provides a compact and structured visual representation of scenes. We study the problem from a physical and topological perspective: a 3D line most naturally emerges as the edge of a finite 3D planar patch. We present LiP-Map, a line-plane joint optimization framework that explicitly models learnable line and planar primitives. This coupling enables accurate and detailed 3D line mapping while maintaining strong efficiency (typically completing a reconstruction in 3 to 5 minutes per scene). LiP-Map pioneers the integration of planar topology into 3D line mapping, not by imposing pairwise coplanarity constraints but by explicitly constructing interactions between plane and line primitives, thus offering a principled route toward structured reconstruction in man-made environments. On more than 100 scenes from ScanNetV2, ScanNet++, Hypersim, 7Scenes, and Tanks\&Temple, LiP-Map improves both accuracy and completeness over state-of-the-art methods. Beyond line mapping quality, LiP-Map significantly advances line-assisted visual localization, establishing strong performance on 7Scenes. Our code is released at https://github.com/calmke/LiPMAP for reproducible research.

Figures (36)

• Figure 1: Comparison of different 3D line maps on scenes of three public datasets. First row: lift 2D detected lines into 3D lines using depths rendered from the GT mesh. Second row: lift 2D detected lines into 3D lines using depths rendered by PlanarSplatting PlanarSplatting2024. Third row: extract planar edges from PlanarSplatting PlanarSplatting2024. Last row: our 3D line maps. 2D line segments are detected by DeepLSD DeepLSD. "lift" means back-project 2D detected lines into 3D using the sensor/predicted depth maps.
• Figure 2: Overview of our proposed reconstruction pipeline, a novel 3D line mapping method by exploring the structural synergies between 3D planes and lines.
• Figure 3: Illustration of surfaces (left) and 3D lines (right) in a ScanNetV2 scene scannet-DaiCSHFN17. The 3D lines align closely with physical surface boundaries, highlighting the strong correlation between the two structures.
• Figure 4: Representation of the 3D rectangular plane with learnable shape parameters.
• Figure 5: Illustration of 2D line segments detected by DeepLSD DeepLSD and their corresponding 1-pixel regions in a single view of the Hypersim Hypersim scene "ai_001_001". Left: the detected 2D lines. Right: the corresponding 1-pixel regions.