Privacy-Preserving Collaborative Medical Image Segmentation Using Latent Transform Networks

TL;DR

This work introduces a privacy-preserving collaborative medical image segmentation framework (PPCMI-SF) designed for heterogeneous medical datasets that combines skip-connected autoencoders for images and masks with a keyed latent transform that applies client-specific orthogonal mixing and permutation to protect latent features before they are shared.

Abstract

Collaborative training across multiple institutions is becoming essential for building reliable medical image segmentation models. However, privacy regulations, data silos, and uneven data availability prevent hospitals from sharing raw scans or annotations, limiting the ability to train generalizable models. Latent-space collaboration frameworks such as privacy-segmentation framework (SF) offer a promising alternative, but such methods still face challenges in segmentation accuracy and vulnerability to latent inversion and membership-inference attacks. This work introduces a privacy-preserving collaborative medical image segmentation framework (PPCMI-SF) designed for heterogeneous medical datasets. The approach combines skip-connected autoencoders for images and masks with a keyed latent transform that applies client-specific orthogonal mixing and permutation to protect latent features before they are shared. A unified mapping network on the server-side performs multi-scale latent-to-latent translation, enabling segmentation inference without exposing raw data. Experiments on four datasets: PSFH ultrasound, ultrasound nerve segmentation, FUMPE CTA, and cardiac MRI show that the proposed PPCMI-SF consistently achieves high Dice scores and improved boundary accuracy, as reflected by lower 95th percentile Hausdorff distance (HD95) and average symmetric surface distance (ASD) compared to the current state-of-the-art and performs competitively with privacy-agnostic baselines. Privacy tests confirm strong resistance to inversion and membership attacks, and the overall system achieves real-time inference with low communication overhead. These results demonstrate that accurate and efficient medical image segmentation can be achieved without compromising data privacy in multi-institution settings.

Figures (8)

• Figure 1: Overview of the proposed privacy-preserving collaborative medical image segmentation framework (PPCMI-SF). (a) Training configuration: Each client trains its own image autoencoder $E_x$–$D_x$ and mask autoencoder $E_y$–$D_y$. The resulting multi-scale latents $\{z_{x1},z_{x2},z_{x3}\}$ and $\{z_{y1},z_{y2},z_{y3}\}$ are secured using the client-keyed transforms $\{T_s(\cdot), T_i(\cdot), T_b(\cdot)\}$ (yellow) before transmission. Upon receiving the protected latents, the server applies the corresponding inverse transforms $\{T_s^{-1},T_i^{-1},T_b^{-1}\}$ (green) to recover the shared-domain latents and trains the unified mapping network $M(\cdot)$ to learn a latent-to-latent translation between unprotected image and mask latents. (b) Inference configuration: The client encodes an input image and applies the forward transforms $\{T_s,T_i,T_b\}$ to protect the latents before sending them to the server. The server again applies the private inverse transforms to map the recovered shared-domain latents through $M(\cdot)$, then re-applies the forward transforms to re-protect the predicted mask latents before returning them to the client. Finally, the client removes the protection, decodes the latents using $D_y$, and obtains the final segmentation mask.
• Figure 2: Visualization of latent distributions before and after the keyed latent transform (KLT).
• Figure 3: Multi-center collaboration framework of the proposed PPCMI-SF. Client institutions train local autoencoders and send only protected latents to the central server, which learns a unified mapping network for segmentation.
• Figure 4: Qualitative reconstruction results from client-specific image and mask autoencoders. The proposed PPCMI-SF yields sharper and more structurally accurate reconstructions across all clients compared to the baseline Privacy-SF method.
• Figure 5: Qualitative comparison of segmentation results on representative PSFH ultrasound images. Rows from top to bottom show the original input image, ground truth annotation, and predicted masks from UNet, GhostNet, FATNet, baseline Privacy-SF, and the proposed PPCMI-SF model, respectively. Fetal head and pubic symphysis regions are highlighted in green and red, respectively.