Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts
Reuben Dorent, Nazim Haouchine, Alexandra Golby, Sarah Frisken, Tina Kapur, William Wells
TL;DR
This work introduces MMHVAE, a hierarchical mixture of multimodal VAEs that synthesizes missing medical images from partial observations by modeling the variational posterior as a mixture of product-of-experts. The approach constructs a deep, multi-level latent representation and a principled fusion mechanism to align latent factors across modalities, while regularizing non-observed distributions with GAN losses. It enables cross-modal synthesis with incomplete training data and demonstrates strong performance on brain MRI and intraoperative ultrasound tasks, including harmonized synthesis, brain tumor segmentation from synthetic iUS, and improved MR–iUS registration. The method yields sharper, more realistic multimodal reconstructions, better downstream task performance, and favorable computational efficiency relative to contemporary unified synthesis models.
Abstract
We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challenges: (i) creating a complex latent representation of multimodal data to generate high-resolution images; (ii) encouraging the variational distributions to estimate the missing information needed for cross-modal image synthesis; (iii) learning to fuse multimodal information in the context of missing data; (iv) leveraging dataset-level information to handle incomplete data sets at training time. Extensive experiments are performed on the challenging problem of pre-operative brain multi-parametric magnetic resonance and intra-operative ultrasound imaging.