Learning to segment anatomy and lesions from disparately labeled sources in brain MRI

TL;DR

This work tackles the challenge of segmenting both healthy brain structures and lesions in MRI when jointly labeled data are scarce. It introduces a dual-path architecture with tissue-focused and lesion-focused encoders, an attention-based fusion head, and an inference-time adaptation guided by meta-learning to maintain healthy-tissue predictions despite lesion disruptions. Training proceeds in three phases—pretraining on task-specific data, meta co-training with synthetic pseudo-lesions, and joint training with real lesion data—enabling joint segmentation from disparately labeled datasets. The method achieves improved joint segmentation performance on BraTS glioblastoma data and demonstrates robustness to domain shifts in an in-house dataset, highlighting its potential for clinically useful brain MRI analysis and treatment monitoring.

Abstract

Segmenting healthy tissue structures alongside lesions in brain Magnetic Resonance Images (MRI) remains a challenge for today's algorithms due to lesion-caused disruption of the anatomy and lack of jointly labeled training datasets, where both healthy tissues and lesions are labeled on the same images. In this paper, we propose a method that is robust to lesion-caused disruptions and can be trained from disparately labeled training sets, i.e., without requiring jointly labeled samples, to automatically segment both. In contrast to prior work, we decouple healthy tissue and lesion segmentation in two paths to leverage multi-sequence acquisitions and merge information with an attention mechanism. During inference, an image-specific adaptation reduces adverse influences of lesion regions on healthy tissue predictions. During training, the adaptation is taken into account through meta-learning and co-training is used to learn from disparately labeled training images. Our model shows an improved performance on several anatomical structures and lesions on a publicly available brain glioblastoma dataset compared to the state-of-the-art segmentation methods.

Figures (16)

• Figure 1: Model overview. U-shaped feature extractors $f_\theta$ and $f_\phi$ takes images with lesion ($x_L$) and lesion free images ($x_A$) images and output segmentations. For joint segmentations ($\hat{y}$), both modalities' feature extractors are used. In the proposed pipeline, first $f_\theta$ is trained with lesion free datasets with tissue labels; and $f_\phi$ is trained with patient datasets and lesion labels to segment task specific datasets. Second, $f_\theta$ is trained in a meta-fashion such that the T1w features can be adapted to presence of lesions in test time. Lastly, $f_\psi$ is added in training such that FLAIR and T1w features can be fused into a joint segmentation output ($\hat{y}$) for images bearing tumors.
• Figure 2: T1 weighted image examples from BraTS dataset (top) and their tissue segmentations with anatomical structure segmentation network after the pretraining stage (bottom). Red boxes highlight areas with lesions. The corresponding parts in the predictions are less accurate than the other parts of the brain (Labels are depicted as follows: red: gray matter, blue: white matter, cyan: ventricle, green: basal ganglia, white: brain stem and magenta: cerebellum).
• Figure 3: Examples of the pseudo-unhealthy dataset, created by pasting the T1w image voxels with lesion label in BraTS dataset, on top of the lesion free dataset examples; to serve the proxy of test images bearing tumors to adapt to in the meta learning setting.
• Figure 4: Comparison of different methods on slices selected from 5 test subjects. First three rows are given in the BraTS dataset, T1w image highlights tissues, whereas in FLAIR lesions are more visible. Tumor subclasses are merged into one class as shown in row 3. Next four rows show competing methods for healthy structure segmentation. We show our model's segmentation in penultimate column, and the last column shows the lineation of classes by radiologist, which is taken as the ground truth for the quantitative analysis. SAMSEG samseg-lesion, SynthSeg billot_synthseg_2021 and VBG kulvbg do not produce tumor segmentation results.
• Figure 5: Qualitative results for SynthSeg and SAMSEG after the original T1w image has transformed with SynthSR. The first row shows the original T1w images from the test image set. The second row shows the output of SynthSR method, where the small lesions should be inpainted. SynthSeg output, and the ground truth lesion annotation overlaid, SAMSEG output and the ground truth lesion annotation overlaid for the same images are shown respectively in the remaining rows.