LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing

TL;DR

The paper tackles semantic segmentation of high-resolution remote sensing data using multimodal inputs RGB, NirRG, and DSM, addressing the limitations of prior two-modality approaches. It introduces LMFNet, a lightweight, weight-sharing multi-branch Vision Transformer framework that fuses arbitrarily many modalities through a novel Multimodal Feature Fusion Module, comprising the Multimodal Feature Fusion Reconstruction Layer and the Multimodal Feature Self-Attention Fusion Layer, followed by an MLP decoder. Empirical results on US3D, ISPRS Potsdam, and Vaihingen show substantial gains in $mIoU$ and $OA$ with a small parameter increase (e.g., $3.72$–$4.22$M), and tri-modal inputs outperform bimodal baselines by a notable margin. The approach demonstrates strong cross-dataset performance, robustness to modality combinations, and supports scalable multimodal fusion with practical implications for land-cover mapping and infrastructure monitoring in remote sensing.

Abstract

Despite the rapid evolution of semantic segmentation for land cover classification in high-resolution remote sensing imagery, integrating multiple data modalities such as Digital Surface Model (DSM), RGB, and Near-infrared (NIR) remains a challenge. Current methods often process only two types of data, missing out on the rich information that additional modalities can provide. Addressing this gap, we propose a novel \textbf{L}ightweight \textbf{M}ultimodal data \textbf{F}usion \textbf{Net}work (LMFNet) to accomplish the tasks of fusion and semantic segmentation of multimodal remote sensing images. LMFNet uniquely accommodates various data types simultaneously, including RGB, NirRG, and DSM, through a weight-sharing, multi-branch vision transformer that minimizes parameter count while ensuring robust feature extraction. Our proposed multimodal fusion module integrates a \textit{Multimodal Feature Fusion Reconstruction Layer} and \textit{Multimodal Feature Self-Attention Fusion Layer}, which can reconstruct and fuse multimodal features. Extensive testing on public datasets such as US3D, ISPRS Potsdam, and ISPRS Vaihingen demonstrates the effectiveness of LMFNet. Specifically, it achieves a mean Intersection over Union ($mIoU$) of 85.09\% on the US3D dataset, marking a significant improvement over existing methods. Compared to unimodal approaches, LMFNet shows a 10\% enhancement in $mIoU$ with only a 0.5M increase in parameter count. Furthermore, against bimodal methods, our approach with trilateral inputs enhances $mIoU$ by 0.46 percentage points.

Figures (13)

• Figure 1: Comparison of different fusion frameworks. In the figure, F represents the fusion module or fusion method.
• Figure 2: Overall framework. To facilitate the input of data into the network, multi-spectral data is separated into RGB and NirRG through false color synthesis; a single-band DSM is converted into RGB by using colormap.
• Figure 3: LMFNet structure. On the left side of the figure is the overall structure, on the upper right side is the structure of the encoder, in the middle right is the structure of the multimodal fusion module, and the bottom right shows the structure of the decoder and classifier.
• Figure 4: Multimodal feature fusion and merge module.
• Figure 5: The structures of MFFR Layer and MFSAF Layer