Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model

TL;DR

Med-2E3 tackles generalization gaps in 3D medical image analysis by jointly encoding 3D volume features $\mathbf{z}_{\text{3D}}$ and per-slice 2D features $\mathbf{z}_{\text{2D}}^j$, then computing slice attention via $s^j = \text{Softmax}(\text{MLP}(\mathbf{z}_{\text{T}}) \cdot \mathbf{z}^j)$ and forming $\mathbf{z}_{\text{I}} = [\mathbf{z}_{\text{3D}}; \mathbf{z}_{\text{2D}}]$ for LLM processing. The model employs a Text-Guided Inter-Slice (TG-IS) module to simulate task-dependent radiologist attention across slices and fuses dual 3D-2D features before generating text with a medical LLM. Experiments on CT-RATE and M3D-Data show Med-2E3 achieving state-of-the-art performance on both medical report generation and VQA tasks, while providing interpretable attention patterns that vary by task and sample. This approach offers a scalable, interpretable framework for multimodal 3D medical understanding and motivates extending to other modalities like MRI and PET in future work.

Abstract

3D medical image analysis is essential for modern healthcare, yet traditional task-specific models are inadequate due to limited generalizability across diverse clinical scenarios. Multimodal large language models (MLLMs) offer a promising solution to these challenges. However, existing MLLMs have limitations in fully leveraging the rich, hierarchical information embedded in 3D medical images. Inspired by clinical practice, where radiologists focus on both 3D spatial structure and 2D planar content, we propose Med-2E3, a 3D medical MLLM that integrates a dual 3D-2D encoder architecture. To aggregate 2D features effectively, we design a Text-Guided Inter-Slice (TG-IS) scoring module, which scores the attention of each 2D slice based on slice contents and task instructions. To the best of our knowledge, Med-2E3 is the first MLLM to integrate both 3D and 2D features for 3D medical image analysis. Experiments on large-scale, open-source 3D medical multimodal datasets demonstrate that TG-IS exhibits task-specific attention distribution and significantly outperforms current state-of-the-art models. The code is available at: https://github.com/MSIIP/Med-2E3

Figures (6)

• Figure 1: Diagnostic workflow of interpreting 3D medical images. Radiologists typically analyze 3D medical images from both global (3D) and local (2D) perspectives, enabling them to focus on spatial structures and planar content. They allocate attention differently to slices based on their content and the specific task requirements.
• Figure 2: Overall framework of Med-2E3. Med-2E3 employs a dual 3D-2D encoder architecture to extract complementary features ($\mathbf{z}_{\text{3D}}$ and $\{\mathbf{z}_{\text{2D}}^j\}$), respectively. Attention scores $\mathbf{s}\in\mathbb{R}^N$, calculated by the TG-IS scoring module, are used to aggregate 2D features $\{\mathbf{z}_{\text{2D}}^j\}$. Aggregated 2D features $\mathbf{z}_{\text{2D}}$ are concatenated with 3D features $\mathbf{z}_{\text{3D}}$. Finally, LLM process the 2D-enhanced 3D medical image features $\mathbf{z}_{\text{I}}$ with text input $\mathbf{x}_{\text{T}}$ to generate the response $\mathbf{x}_{\text{R}}$.
• Figure 3: TG-IS scoring module. 3D features $\mathbf{z}_{\text{3D}}$ and 2D features $\{\mathbf{z}_{\text{2D}}^j\}$ are fused to derive the key features $\mathbf{z}^j$ for each slice. Attention scores $\mathbf{s} \in \mathbb{R}^N$ are computed by calculating the correlation $\{\mathbf{s}_{\text{r}}^j\}$ with the task instruction feature $\mathbf{z}_{\text{T}}$, followed by the softmax function for normalization.
• Figure 4: Visualization of two CT volumes and their corresponding question-answer pairs. To improve clarity, 8 slices were uniformly sampled from the full CT sequence. Questions 1 and 5 correspond to report generation tasks, while questions 2, 3, 6, 7 are closed-ended VQA tasks (multiple-choice), and questions 4 and 8 are open-ended VQA tasks (free-form). Keywords in each question are highlighted in bold. The red dashed line in the score distribution represents the average attention score ($1/32$). Distinct attention distributions are observed for different questions on the same CT volume, while similar distribution patterns emerge for the same question posed in different formats (multiple-choice and free-form). This demonstrates that Med-2E3 effectively captures essential task-relevant information.
• Figure 5: Ablation study.