Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks

TL;DR

The paper tackles the challenge of fusing multimodal data with heterogeneous dimensionality to support localization tasks like segmentation. It introduces a projective network framework that projects modality features into a common $n$-dimensional space using modules for feature extraction, projection, and fusion, and proposes two fusion strategies: Late Fusion and Multiscale Fusion. The methods are validated on GA and RBV segmentation in multimodal retinal imaging, where they consistently outperform state-of-the-art monomodal baselines, particularly under limited labeled data. The work demonstrates robustness to noise and data scarcity, discusses practical limitations such as modality registration, and provides publicly available code for broader adoption in multimodal medical image analysis.

Abstract

The use of multimodal imaging has led to significant improvements in the diagnosis and treatment of many diseases. Similar to clinical practice, some works have demonstrated the benefits of multimodal fusion for automatic segmentation and classification using deep learning-based methods. However, current segmentation methods are limited to fusion of modalities with the same dimensionality (e.g., 3D+3D, 2D+2D), which is not always possible, and the fusion strategies implemented by classification methods are incompatible with localization tasks. In this work, we propose a novel deep learning-based framework for the fusion of multimodal data with heterogeneous dimensionality (e.g., 3D+2D) that is compatible with localization tasks. The proposed framework extracts the features of the different modalities and projects them into the common feature subspace. The projected features are then fused and further processed to obtain the final prediction. The framework was validated on the following tasks: segmentation of geographic atrophy (GA), a late-stage manifestation of age-related macular degeneration, and segmentation of retinal blood vessels (RBV) in multimodal retinal imaging. Our results show that the proposed method outperforms the state-of-the-art monomodal methods on GA and RBV segmentation by up to 3.10% and 4.64% Dice, respectively.

Figures (11)

• Figure 1: Our proposed framework defines a novel fusion approach that extracts and projects the features of all modalities into the feature space of the modality with the lowest dimensionality ($n$), so that they can be employed for localization tasks in the common ($n$-dimensional, $n$D) subspace.
• Figure 2: Illustration of the proposed framework, consisting of 3 modules: feature extractor (FE), projective feature extractor (PFE), and feature fusion module (FFM). The FE extracts $n$D features from $n$D data, and the PFE extracts $n$D features from $q$D data by projecting them to the $n$D feature space. Then, the FFM processes the $n$D features extracted from the different modalities to obtain the final $m$D prediction, where $m \leq n$.
• Figure 3: Illustration of the Late Fusion approach. The features extracted from different encoder-decoder modules for each modality are concatenated and processed by a simple convolutional block.
• Figure 4: Illustration of the Multiscale Fusion approach. The features extracted from different encoders for each modality are fused at multiple scales and processed by a single decoder.
• Figure 5: Proposed FCNN for multimodal fusion. For Late Fusion, the Image branch is an encoder-decoder architecture, and its output () is concatenated with the output of the Volume branch before the final convolution. For Multiscale Fusion, only the encoder of the Image branch is used, and its features at different scales () are concatenated with the features of the encoder from the Volume branch and fed to a common decoder. Before concatenation, all features are resized to the minimum feature size using adaptive max-pooling. The number of feature maps is indicated in bold.