HyperPointFormer: Multimodal Fusion in 3D Space with Dual-Branch Cross-Attention Transformers

TL;DR

The paper tackles the limitation of 2D fusion in multimodal remote sensing by introducing HyperPointFormer, a fully 3D fusion framework that processes lidar-based geometry and spectral data in two dedicated Transformer branches. A novel CrossPointAttention module enables bidirectional, multi-scale fusion by computing cross-attention between branches in latent space, improving per-point classification in 3D while supporting projection to 2D maps. Across DFC2018, Vaihingen 3D, and DFC2019, the method achieves competitive 3D results, with notable gains over strong baselines and evidence that mid-level fusion and CPA are crucial. The work highlights 3D fusion’s practical benefits for urban scene analysis and provides datasets and code to promote further research in 3D multimodal fusion.

Abstract

Multimodal remote sensing data, including spectral and lidar or photogrammetry, is crucial for achieving satisfactory land-use / land-cover classification results in urban scenes. So far, most studies have been conducted in a 2D context. When 3D information is available in the dataset, it is typically integrated with the 2D data by rasterizing the 3D data into 2D formats. Although this method yields satisfactory classification results, it falls short in fully exploiting the potential of 3D data by restricting the model's ability to learn 3D spatial features directly from raw point clouds. Additionally, it limits the generation of 3D predictions, as the dimensionality of the input data has been reduced. In this study, we propose a fully 3D-based method that fuses all modalities within the 3D point cloud and employs a dedicated dual-branch Transformer model to simultaneously learn geometric and spectral features. To enhance the fusion process, we introduce a cross-attention-based mechanism that fully operates on 3D points, effectively integrating features from various modalities across multiple scales. The purpose of cross-attention is to allow one modality to assess the importance of another by weighing the relevant features. We evaluated our method by comparing it against both 3D and 2D methods using the 2018 IEEE GRSS Data Fusion Contest (DFC2018) dataset. Our findings indicate that 3D fusion delivers competitive results compared to 2D methods and offers more flexibility by providing 3D predictions. These predictions can be projected onto 2D maps, a capability that is not feasible in reverse. Additionally, we evaluated our method on different datasets, specifically the ISPRS Vaihingen 3D and the IEEE 2019 Data Fusion Contest. Our code will be published here: https://github.com/aldinorizaldy/hyperpointformer.

Figures (14)

• Figure 1: 3D prediction result of the multimodal point clouds.
• Figure 2: Classical fusion methods: (a) early, (b) late fusion, and (c) our multi-scale features fusion.
• Figure 3: The overall architecture of HyperPointFormer and the CrossPointAttention (CPA) module. HyperPointFormer takes XYZ coordinates ($N \times 3$) and spectral information ($N \times b$) of multimodal lidar point clouds at different branches, where $N$ is the number of points and $b$ is the number of spectral bands. It utilizes Vector Self-Attention as the backbone encoders, while Farthest Point Sampling (FPS) is used to downsample the number of points at each layer. The CPA module, which is explained in more detail in Section \ref{['sec:fusion']}, takes encoded features from different modalities at various scales and fuses them using cross-attention mechanism. Additionally, skip connections are used in the upsampling stages to enhance the detail in the segmentation outputs. The output is a predicted label for each 3D point ($N \times C$), where $C$ represents the number of classes.
• Figure 4: The actual Train and Test split of the DFC2018 dataset.
• Figure 5: Multimodal 3D point clouds data, rendered with different features: (a) RGB, (b) hyperspectral, and (c) elevation.