Physics Informed Generative AI Enabling Labour Free Segmentation For Microscopy Analysis

TL;DR

This work tackles the data scarcity and sim-to-real gap in microscopy segmentation by blending physics-based phase-field simulations with CycleGAN-driven appearance translation to SEM-like images. A U-Net trained exclusively on these synthetic, label-rich images achieves strong generalization to unseen experimental SEM data, eliminating the need for manual annotations. Comprehensive validation shows the synthetic data occupy the core real data manifold (via SSIM, t-SNE, and entropy analyses) and yield high segmentation metrics (IoU and Boundary F1 near 0.9). The approach enables automated microstructure characterization and dynamic analyses (e.g., grain growth) at scale, offering a robust pathway toward high-throughput, annotation-free materials discovery and design, with future potential for 3D and multi-material extensions.

Abstract

Semantic segmentation of microscopy images is a critical task for high-throughput materials characterisation, yet its automation is severely constrained by the prohibitive cost, subjectivity, and scarcity of expert-annotated data. While physics-based simulations offer a scalable alternative to manual labelling, models trained on such data historically fail to generalise due to a significant domain gap, lacking the complex textures, noise patterns, and imaging artefacts inherent to experimental data. This paper introduces a novel framework for labour-free segmentation that successfully bridges this simulation-to-reality gap. Our pipeline leverages phase-field simulations to generate an abundant source of microstructural morphologies with perfect, intrinsically-derived ground-truth masks. We then employ a Cycle-Consistent Generative Adversarial Network (CycleGAN) for unpaired image-to-image translation, transforming the clean simulations into a large-scale dataset of high-fidelity, realistic SEM images. A U-Net model, trained exclusively on this synthetic data, demonstrated remarkable generalisation when deployed on unseen experimental images, achieving a mean Boundary F1-Score of 0.90 and an Intersection over Union (IOU) of 0.88. Comprehensive validation using t-SNE feature-space projection and Shannon entropy analysis confirms that our synthetic images are statistically and featurally indistinguishable from the real data manifold. By completely decoupling model training from manual annotation, our generative framework transforms a data-scarce problem into one of data abundance, providing a robust and fully automated solution to accelerate materials discovery and analysis.

Figures (6)

• Figure 1: Schematic of the proposed framework for labour-free semantic segmentation. (1) Data Acquisition: An unlabeled target domain, $X$, is formed from preprocessed experimental SEM images. (2) Phase-Field Simulation: A labeled source domain, $Y$, is created by simulating grain growth to generate diverse microstructural morphologies ($y$) and their corresponding perfect, binarized segmentation masks ($y_{\text{mask}}$). (3) Image-to-Image Translation: A CycleGAN is trained on the unpaired domains $X$ and $Y$. The generator $F: Y \to X$ learns to translate clean simulations into realistic SEM-style images ($\hat{x}$) while preserving the underlying structure. (4) Semantic Segmentation: A U-Net is trained exclusively on the synthetic dataset composed of generated images and their original masks $(\hat{x}, y_{\text{mask}})$. The trained model can then be directly deployed on real SEM images for annotation-free segmentation.
• Figure 2: Qualitative comparison between real experimental SEM images and synthetic microstructures generated by the trained CycleGAN. (a) A selection of real SEM images, showcasing typical contrast and texture. (b) Synthetic microstructures generated by the $F: Y \to X$ generator. The model successfully imparts a realistic SEM appearance onto the clean, simulated grain structures while preserving the underlying morphology. The high average SSIM score of 0.89 calculated for these pairs quantitatively supports the visual similarity.
• Figure 3: t-SNE Visualization of Feature Space Across All Image Domains. This figure visualizes the high-dimensional feature space of all image types projected into two dimensions using t-SNE. Each point represents an image patch, color-coded by its origin (SEM, Phase Field, or No Texture) and symbol-coded by its type (Original, Segmentation, or Ground Truth). The plot two reveals distinct macro-clusters.
• Figure 4: Segmentation Final Results using different datasets showcasing the superiority of our proposed labour-free model
• Figure 5: Automated Microstructure Characterisation. The figure illustrates the complete workflow for extracting key morphological parameters from an unseen experimental SEM image. (a) The original micrograph is input into the trained U-Net model, which generates (b) a high-fidelity semantic segmentation mask delineating the grain boundaries. (c) This binary mask is subsequently processed to create an instance map, where each distinct grain is uniquely identified and labelled. From this map, quantitative data is extracted to generate statistical distributions for critical microstructural features, including (d) grain area and (e) grain circularity. This automated pipeline enables rapid, objective, and reproducible materials characterization directly from experimental images.