Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

U-Harmony: Enhancing Joint Training for Segmentation Models with Universal Harmonization

Weiwei Ma, Xiaobing Yu, Peijie Qiu, Jin Yang, Pan Xiao, Xiaoqi Zhao, Xiaofeng Liu, Tomo Miyazaki, Shinichiro Omachi, Yongsong Huang

TL;DR

The paper tackles domain shift in cross-institution 3D medical image segmentation by introducing Universal Harmonization (U-Harmony), a two-stage per-sample feature harmonization and restoration module that aligns features while preserving domain-specific knowledge, coupled with a dataset-free domain-gated head for inference. This approach enables a single model to jointly learn from heterogeneous modalities and annotation schemes, demonstrated across three brain lesion datasets with consistent performance gains over strong baselines. Key contributions include the harmonization-restoration pipeline, the dataset-free gating mechanism, and substantial improvements in cross-domain generalization, suggesting robust, adaptable segmentation in real-world clinical settings without requiring domain labels at test time. The findings indicate that U-Harmony effectively mitigates domain and modality shifts while maintaining dataset-specific information, setting a new benchmark for universal joint training in 3D medical image segmentation.

Abstract

In clinical practice, medical segmentation datasets are often limited and heterogeneous, with variations in modalities, protocols, and anatomical targets across institutions. Existing deep learning models struggle to jointly learn from such diverse data, often sacrificing either generalization or domain-specific knowledge. To overcome these challenges, we propose a joint training method called Universal Harmonization (U-Harmony), which can be integrated into deep learning-based architectures with a domain-gated head, enabling a single segmentation model to learn from heterogeneous datasets simultaneously. By integrating U-Harmony, our approach sequentially normalizes and then denormalizes feature distributions to mitigate domain-specific variations while preserving original dataset-specific knowledge. More appealingly, our framework also supports universal modality adaptation, allowing the seamless learning of new imaging modalities and anatomical classes. Extensive experiments on cross-institutional brain lesion datasets demonstrate the effectiveness of our approach, establishing a new benchmark for robust and adaptable 3D medical image segmentation models in real-world clinical settings.

U-Harmony: Enhancing Joint Training for Segmentation Models with Universal Harmonization

TL;DR

The paper tackles domain shift in cross-institution 3D medical image segmentation by introducing Universal Harmonization (U-Harmony), a two-stage per-sample feature harmonization and restoration module that aligns features while preserving domain-specific knowledge, coupled with a dataset-free domain-gated head for inference. This approach enables a single model to jointly learn from heterogeneous modalities and annotation schemes, demonstrated across three brain lesion datasets with consistent performance gains over strong baselines. Key contributions include the harmonization-restoration pipeline, the dataset-free gating mechanism, and substantial improvements in cross-domain generalization, suggesting robust, adaptable segmentation in real-world clinical settings without requiring domain labels at test time. The findings indicate that U-Harmony effectively mitigates domain and modality shifts while maintaining dataset-specific information, setting a new benchmark for universal joint training in 3D medical image segmentation.

Abstract

In clinical practice, medical segmentation datasets are often limited and heterogeneous, with variations in modalities, protocols, and anatomical targets across institutions. Existing deep learning models struggle to jointly learn from such diverse data, often sacrificing either generalization or domain-specific knowledge. To overcome these challenges, we propose a joint training method called Universal Harmonization (U-Harmony), which can be integrated into deep learning-based architectures with a domain-gated head, enabling a single segmentation model to learn from heterogeneous datasets simultaneously. By integrating U-Harmony, our approach sequentially normalizes and then denormalizes feature distributions to mitigate domain-specific variations while preserving original dataset-specific knowledge. More appealingly, our framework also supports universal modality adaptation, allowing the seamless learning of new imaging modalities and anatomical classes. Extensive experiments on cross-institutional brain lesion datasets demonstrate the effectiveness of our approach, establishing a new benchmark for robust and adaptable 3D medical image segmentation models in real-world clinical settings.
Paper Structure (10 sections, 3 equations, 3 figures, 3 tables)

This paper contains 10 sections, 3 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Joint training with U-Harmony
  4. Domain-Gated Head for Dataset-free Inference
  5. Experiments
  6. Experimental setup
  7. Experimental results
  8. Conclusion
  9. COMPLIANCE WITH ETHICAL STANDARDS
  10. Acknowledgement

Figures (3)

  • Figure 1: The overview of our U-Harmony framework and domain-gated head network.
  • Figure 2: The comparison showcases the segmentation results on Original datasets UCSF-BMSR and BrainMetShare, transitioning to joint tasks (UCSF-BMSR + BrainMet). The U-Harmony method achieves superior boundary delineation and adaptive accuracy across stages compared to CVCL.
  • Figure 3: The comparison with ablation showcases the segmentation results on Original datasets UCSF-BMSR and BraTS-MENTs 2023, transitioning to joint tasks (UCSF-BMSR + BraTS-MENTs 2023). The highlighted regions demonstrate U-Harmony's robustness in handling domain shifts and extra classes.