Improving 3D deep learning segmentation with biophysically motivated cell synthesis

TL;DR

The paper addresses the challenge of obtaining high-quality 3D ground-truth data for cell segmentation in dense spheroid cultures by introducing biophysically motivated data synthesis. It combines a 3D Cellular Potts Model–driven simulation with CycleGAN/GAN-based generation to create coherent membrane and nuclei signals, including a novel training scheme that yields matching labels. Quantitative results show that biophysically informed synthetic data can outperform manual annotations and pretrained models in segmentation performance, and kernel-based metrics confirm the closer structural alignment to real data. The approach reduces manual labeling effort and offers scalable, multi-channel training data, with potential applicability to other imaging modalities and markers in 3D cell culture analysis.

Abstract

Biomedical research increasingly relies on 3D cell culture models and AI-based analysis can potentially facilitate a detailed and accurate feature extraction on a single-cell level. However, this requires for a precise segmentation of 3D cell datasets, which in turn demands high-quality ground truth for training. Manual annotation, the gold standard for ground truth data, is too time-consuming and thus not feasible for the generation of large 3D training datasets. To address this, we present a novel framework for generating 3D training data, which integrates biophysical modeling for realistic cell shape and alignment. Our approach allows the in silico generation of coherent membrane and nuclei signals, that enable the training of segmentation models utilizing both channels for improved performance. Furthermore, we present a new GAN training scheme that generates not only image data but also matching labels. Quantitative evaluation shows superior performance of biophysical motivated synthetic training data, even outperforming manual annotation and pretrained models. This underscores the potential of incorporating biophysical modeling for enhancing synthetic training data quality.

Figures (8)

• Figure 1: Presented workflow for generation of 3D synthetic images featuring nuclei and membrane signals. 3D cell cultures are imaged using confocal microscopy. Afterwards, parameters are extracted based on the recorded images which are then utilized during cell simulation to generate synthetic cell border images. Finally, synthetic nuclei and membrane images are generated on the basis of the simulated cell border image.
• Figure 2: Biophysically-driven 3D spheroid simulations from real image data reveal cell morphology variability across Monte Carlo steps (MCS), while preserving the distribution of morphological features throughout the spheroid. A 3D maximum projection of membrane marker SiR-actin (grey,far left), overlay with label masks (second from the left), and snapshots of the corresponding CPM simulation of morphological changes at the start (middle, 0 MCS), middle (second from the right, 400 MCS) and end (far right, 1000 MCS) of the simulation. B Zoom images of the simulation steps shown in (a). Scale bars: (a) 25µm, (b) 15µm.
• Figure 3: Comparison of morphological feature distributions at the start and end of simulations reflecting changes in cell morphology, for the best and worst parameter sets according to the metric $m$, Eq. \ref{['eq:metric_paramscan']}. Data is derived from a manually segmented image patch. Individual Wasserstein distances, quantifying the variations between the simulation's start and end for each feature, are marked above the plots for the best (blue, $\lambda_V=10.0$, $\lambda_A = 0.001$, $J^\textrm{(c-c)} = 2.0$, $J^\textrm{(c-m)} = 55.0$) and worst (orange, $\lambda_V=0.001$, $\lambda_A = 10.0$, $J^\textrm{(c-c)} = 10.0$, $J^\textrm{(c-m)} = 10.0$) parameter sets. Boxes indicate quartiles of data, while whiskers encompass all values within 1.5 times of the IQR, and outliers are indicated as single data points.
• Figure 4: Comparison of real and synthetic images. A shows image slices of real and synthetic 3D nuclei images. SimOptiGAN uses a random process for nuclei arrangement, while SimOptiGAN+, Mem2NucGAN-P and Mem2NucGAN-U incorporate biophysical modeling for a realistic arrangement. B shows image slices of real and synthetic membrane signals. The synthetic membrane signal is generated based on the same simulated cell borders used in the nuclei synthesis methods SimOptiGAN+, Mem2NucGAN-P and Mem2NucGAN-U. Consequently, the membrane signal exhibited a consistent cell arrangement, as demonstrated by the overlay of synthetic membrane with nuclei generated using SimOptiGAN+. C shows a preview of naively generated data used as worst example for the KID evaluation in D. D comparison of synthetic nuclei images with real counterparts based on the Kernel Inception Distance (KID). Lower scores represent a greater similarity between real and synthetic signals.
• Figure 5: A SEG and DET segmentation scores of nuclei segmentation models trained with different type of training data. Scores can range from zero (worst-possible) to one (best-possible). Three manual corrected image patches of different image regions serve as test data. Maximum indicates a model that was trained on the test data and is considered an upper boundary. Dark blue color indicates training data generated by manual annotation. The nuclei model provided by Cellpose is depicted in light blue color. Dark and light orange colors indicate the pure use of synthetic training data generated with physical simulation based and GAN-based approaches, respectively. The error bars represent the standard deviation of model performance across the three ground truth patches. B qualitative comparison of nuclei segmentation results. Representative single optical sections of ground truth patches are shown for enhanced clarity. The first and second column display the raw image signal and its corresponding ground truth, while subsequent columns show the segmentation masks obtained with the segmentation models. Additionally, the last row visualizes the DET related errors of the third row, including false-negative, false-positive, and required splitting operations. A complete visualization of DET errors across all ground truth patches is given in the supplementary material (Fig. A2).