Two-Stage Approach for Brain MR Image Synthesis: 2D Image Synthesis and 3D Refinement

TL;DR

This work tackles the challenge of brain MRI segmentation when one or more MR sequences are missing by introducing a two-stage framework. The first stage performs 2D slice-based MR image synthesis with intensity encoding to mitigate inter-slice stripe artifacts, while the second stage uses a Refiner to exploit full 3D context and improve tumor representation through a voxel-wise cross-attention mechanism and a 3D-aware decoder. The approach yields improved Dice scores for tumor sub-regions and achieves top performance in the BraSyn 2024 challenge, demonstrating practical impact for clinical workflows where complete multimodal MRI data are unavailable. The combination of intensity-aware 2D synthesis and 3D refinement offers a promising path to robust, segmentation-friendly MRI synthesis under real-world computational constraints.

Abstract

Despite significant advancements in automatic brain tumor segmentation methods, their performance is not guaranteed when certain MR sequences are missing. Addressing this issue, it is crucial to synthesize the missing MR images that reflect the unique characteristics of the absent modality with precise tumor representation. Typically, MRI synthesis methods generate partial images rather than full-sized volumes due to computational constraints. This limitation can lead to a lack of comprehensive 3D volumetric information and result in image artifacts during the merging process. In this paper, we propose a two-stage approach that first synthesizes MR images from 2D slices using a novel intensity encoding method and then refines the synthesized MRI. The proposed intensity encoding reduces artifacts when synthesizing MRI on a 2D slice basis. Then, the \textit{Refiner}, which leverages complete 3D volume information, further improves the quality of the synthesized images and enhances their applicability to segmentation methods. Experimental results demonstrate that the intensity encoding effectively minimizes artifacts in the synthesized MRI and improves perceptual quality. Furthermore, using the \textit{Refiner} on synthesized MRI significantly improves brain tumor segmentation results, highlighting the potential of our approach in practical applications.

Figures (4)

• Figure 1: The structure of modality infuser with intensity encoding. The target intensity and target modality are jointly encoded. In the training phase, the target intensity is directly computed from the target MR sequence. In the inference phase, the intensity prior is pre-computed from the training dataset and the target intensity is sampled from this distribution.
• Figure 2: The structure of Refiner. The feature representation of synthesized MR image, $\hat{F}_{m_i}$, and of all available MR image, $F_{m_j}$, are obtained using the corresponding encoder. Then, voxel-wise cross-attention is performed between $\hat{F}_{m_i}$ and each of $F_{m_j}$ sequentially. Finally, the refined feature representation is decoded into the image space and added to the input MR image $\hat{X}_{m_i}$ to produce the final output $\hat{X}_{m_i}'$.
• Figure 3: An example of 2D-based MR image synthesized results when T1 MRI is missing. A, C, and S represent axial, coronal and sagittal slices, respectively.
• Figure 4: An example of synthesized output from the HF-GAN with Intensity Encoding (mean) and the refined output from the Refiner. A, C, and S stand for axial, coronal, and sagittal slices, respectively, and next to each MR image is a segmentation prediction mask obtained using FeTS model.