Multi-modal and Multi-view Fundus Image Fusion for Retinopathy Diagnosis via Multi-scale Cross-attention and Shifted Window Self-attention

TL;DR

This work tackles the challenge of retinopathy diagnosis from multi-modal fundus data (CFP and FFA) across multiple views by modeling long-range and multi-scale dependencies. It introduces MRDF, a multi-task transformer-based framework that fuses information with two dedicated modules: MFSWFM for efficient multi-view fusion using shifted window self-attention, and CFFT for cross-modal fusion via multi-scale cross-attention. The encoder-decoder design supports retinopathy classification and symptom report generation, with extensive ablations showing the value of LRCL, position embeddings, and multi-view embeddings. Experiments on private SFPD and MSRD datasets demonstrate superior accuracy and report-generation quality, suggesting a practical impact on ophthalmologists' workflow and potential extension to 3D multi-modal imaging.

Abstract

The joint interpretation of multi-modal and multi-view fundus images is critical for retinopathy prevention, as different views can show the complete 3D eyeball field and different modalities can provide complementary lesion areas. Compared with single images, the sequence relationships in multi-modal and multi-view fundus images contain long-range dependencies in lesion features. By modeling the long-range dependencies in these sequences, lesion areas can be more comprehensively mined, and modality-specific lesions can be detected. To learn the long-range dependency relationship and fuse complementary multi-scale lesion features between different fundus modalities, we design a multi-modal fundus image fusion method based on multi-scale cross-attention, which solves the static receptive field problem in previous multi-modal medical fusion methods based on attention. To capture multi-view relative positional relationships between different views and fuse comprehensive lesion features between different views, we design a multi-view fundus image fusion method based on shifted window self-attention, which also solves the computational complexity of the multi-view fundus fusion method based on self-attention is quadratic to the size and number of multi-view fundus images. Finally, we design a multi-task retinopathy diagnosis framework to help ophthalmologists reduce workload and improve diagnostic accuracy by combining the proposed two fusion methods. The experimental results of retinopathy classification and report generation tasks indicate our method's potential to improve the efficiency and reliability of retinopathy diagnosis in clinical practice, achieving a classification accuracy of 82.53\% and a report generation BlEU-1 of 0.543.

Figures (7)

• Figure Fig. 1: An example of multi-modal and multi-view fundus images with symptom reports. The diagnostic report is based on five fundus images with two modalities, where the top four images (Vl, V2, V3, V4) are FFA images with different views, and the image in the bottom left corner is a CFP image. Each report contains symptom and pathology sections, where the phrases marked in red are observations from the CFP image, and the phrases marked in blue are observations from these FFA images. Multi-modal and multi-view fundus images contain modality sequence relationships and view sequence relationships, compared with single-modal images.
• Figure Fig. 2: Comparison of different attention mechanisms in previous multi-modal medical image fusion methods and our CFFT. The size of the box indicates the receptive field size of the corresponding token embedding. These arrows indicate the correspondence relationships between inter-modal token embeddings.
• Figure Fig. 3: (a) The architecture of the proposed MRDF. The proposed framework is a network structure that can handle any number of modalities and views by expanding the backbone. For simplicity, only two modalities and two perspectives are shown; (b) The architecture of Transformer; (c) The architecture of MFSWFM.
• Figure Fig. 4: An illustration of the multi-view shifted operation for computing self-attention in the proposed MFSWFM architecture. In layer $l$ (up), these local windows with self-attention are applied in each view. In the next $l + 1$ (down), the window partitioning is shifted and bridges the token embedding of different views, providing connections among different views.
• Figure Fig. 5: (a) The architecture of our proposed CFFT is presented, which contains three MMCAM parts with different parameters $r$; (b) The architecture of MMCAM; (c) The architecture of MCA; (d) A schematic diagram of receptive fields with different parameters $r$ in the proposed MMCAM.