Multimodal Signal Processing For Thermo-Visible-Lidar Fusion In Real-time 3D Semantic Mapping
Jiajun Sun, Yangyi Ou, Haoyuan Zheng, Chao yang, Yue Ma
TL;DR
This work tackles the challenge of generating semantically enriched 3D maps for autonomous navigation by integrating LiDAR-IMU geometry with pixel-level fusion of visible and thermal imagery. The authors introduce a tri-modal fusion SLAM framework where a LiDAR-IMU backbone supports real-time projection of fused visual-thermal textures onto a 3D map, enabling heat-source localization via semantic labeling. Key contributions include a targetless extrinsic calibration method, a large-scale system for temperature-texture-geometry joint modeling, and field experiments on large structures demonstrating robustness under sensor disturbances. The results show real-time operation with accurate thermal anomaly localization, offering significant utility for disaster assessment and industrial maintenance through enhanced situational awareness and proactive response. The fused texture is given by $I_{fused} = w \cdot I_{thermal} + (1-w) \cdot I_{visible}$, the state is $x = [p, v, q, b_{a}, b_{g}, g]^T$ with IMU dynamics $a^{w} = R(a^{b} - b_{a} - n_{a}) - g$, highlighting a rigorous integration of geometry, texture, and temperature semantics into 3D mapping.
Abstract
In complex environments, autonomous robot navigation and environmental perception pose higher requirements for SLAM technology. This paper presents a novel method for semantically enhancing 3D point cloud maps with thermal information. By first performing pixel-level fusion of visible and infrared images, the system projects real-time LiDAR point clouds onto this fused image stream. It then segments heat source features in the thermal channel to instantly identify high temperature targets and applies this temperature information as a semantic layer on the final 3D map. This approach generates maps that not only have accurate geometry but also possess a critical semantic understanding of the environment, making it highly valuable for specific applications like rapid disaster assessment and industrial preventive maintenance.