SegHeD: Segmentation of Heterogeneous Data for Multiple Sclerosis Lesions with Anatomical Constraints

TL;DR

This work addresses automated segmentation of MS brain lesions from heterogeneous MRI data collected across sites, formats, and annotations. It introduces SegHeD, a universal multi-dataset multi-task framework based on a 3D V-Net that outputs $p_a^{t1}$, $p_a^{t2}$, $p_n^{t2}$, and $p_v^{t2}$ from inputs $(x^{t1},x^{t2},y_a^{t1},x_{wm}^{t2})$, enabling all-lesion, new-lesion, and vanishing-lesion segmentation. SegHeD integrates longitudinal, volumetric, and spatial constraints via losses $ ext{L}_{Long}$, $ ext{L}_{Vol}$, and $ ext{L}_{Spat}$, with curriculum learning and a sliding-window strategy for additional timepoints, and can accommodate missing inputs. Empirical evaluation on five MS datasets shows state-of-the-art performance for all-lesion segmentation, competitive performance for new-lesion segmentation, and the first reported results for vanishing-lesion segmentation, alongside improved temporal consistency. This approach enables scalable, multi-site MS imaging analyses with diverse annotations, reducing labeling burden and broadening clinical applicability.

Abstract

Assessment of lesions and their longitudinal progression from brain magnetic resonance (MR) images plays a crucial role in diagnosing and monitoring multiple sclerosis (MS). Machine learning models have demonstrated a great potential for automated MS lesion segmentation. Training such models typically requires large-scale high-quality datasets that are consistently annotated. However, MS imaging datasets are often small, segregated across multiple sites, with different formats (cross-sectional or longitudinal), and diverse annotation styles. This poses a significant challenge to train a unified MS lesion segmentation model. To tackle this challenge, we present SegHeD, a novel multi-dataset multi-task segmentation model that can incorporate heterogeneous data as input and perform all-lesion, new-lesion, as well as vanishing-lesion segmentation. Furthermore, we account for domain knowledge about MS lesions, incorporating longitudinal, spatial, and volumetric constraints into the segmentation model. SegHeD is assessed on five MS datasets and achieves a high performance in all, new, and vanishing-lesion segmentation, outperforming several state-of-the-art methods in this field.

Figures (4)

• Figure 1: Visualisation of the proposed framework. SegHeD learns from heterogeneous datasets varying in image and label formats. It can analyse cross-sectional and longitudinal data, and segment all (red), new (green), and vanishing (dashed blue) lesions.
• Figure 2: Qualitative comparison of all-lesion (top row) and new-lesion (bottom row) segmentation performance. Yellow regions denote false positive segmentations, whereas cyan regions denote false negative segmentations.
• Figure 3: SegHeD is capable of simultaneous multi-task segmentation (Rows 3 to 6). Some tasks do not show new/vanishing-lesions predictions as they are not present at the given slice. "Not available" denotes no ground truth annotation for comparison. A: Dataset where all-lesion labels are available for first and second timepoints. B: Dataset where first timepoint all-lesion label and second timepoint new-lesion label are available. C: Dataset where second timepoint vanishing-lesion label is available.
• Figure 4: Predicted lesion volumes across four timepoints for two test subjects. SegHeD (blue) predictions are temporally more consistent with the ground truth (black), compared to competing methods. The $\rho$ value for each method indicates its Pearson's correlation coefficient with the ground truth, the higher the better.