DiffRenderGAN: Addressing Training Data Scarcity in Deep Segmentation Networks for Quantitative Nanomaterial Analysis through Differentiable Rendering and Generative Modelling

TL;DR

DiffRenderGAN tackles training data scarcity in nanomaterial segmentation by marrying differentiable rendering with GANs to produce annotated synthetic microscopy data. It optimizes rendering parameters within a controllable virtual scene of nanoparticle meshes to generate realistic, labeled images without manual annotation, using $I_r = f_r(Θ)$ and $I_{ ext{synth}} = G(φ_i) = f_{ ext{noise}}(f_r(f_fcn(φ_i), θ_{ ext{other}}), σ)$. Evaluations across TiO$_2$ and SiO$_2$ in HIM, TiO$_2$ in SEM, and AgNW SEM show that synthetic data from DiffRenderGAN can match or surpass prior synthetic approaches and substantially narrow the domain gap to real data, enabling effective segmentation with fewer real annotations. The framework offers a scalable, semi-automated path to multimodal nanomaterial analysis with broad potential for extension to additional imaging modalities and material systems.

Abstract

Nanomaterials exhibit distinctive properties governed by parameters such as size, shape, and surface characteristics, which critically influence their applications and interactions across technological, biological, and environmental contexts. Accurate quantification and understanding of these materials are essential for advancing research and innovation. In this regard, deep learning segmentation networks have emerged as powerful tools that enable automated insights and replace subjective methods with precise quantitative analysis. However, their efficacy depends on representative annotated datasets, which are challenging to obtain due to the costly imaging of nanoparticles and the labor-intensive nature of manual annotations. To overcome these limitations, we introduce DiffRenderGAN, a novel generative model designed to produce annotated synthetic data. By integrating a differentiable renderer into a Generative Adversarial Network (GAN) framework, DiffRenderGAN optimizes textural rendering parameters to generate realistic, annotated nanoparticle images from non-annotated real microscopy images. This approach reduces the need for manual intervention and enhances segmentation performance compared to existing synthetic data methods by generating diverse and realistic data. Tested on multiple ion and electron microscopy cases, including titanium dioxide (TiO$_2$), silicon dioxide (SiO$_2$)), and silver nanowires (AgNW), DiffRenderGAN bridges the gap between synthetic and real data, advancing the quantification and understanding of complex nanomaterial systems.

Figures (15)

• Figure 1: Addressing Training Data Scarcity in Deep Segmentation Networks for Quantitative Nanomaterial Analysis through Synthetic Data Generation. Our contribution aims to address three primary objectives: (1) to present an image synthesis method applicable across various microscopy modalities for the analysis of materials with diverse morphologies, (2) to minimize the need for expert intervention, and (3) to reduce or eliminate the representativeness gap between synthetic and real data, as observed in previous studies, enabling more efficient training of deep segmentation networks for improved analysis of complex nanomaterial systems. It is important to note that our goal is not to generate a physically accurate simulation of materials but rather to conduct a simulation that produces images capturing the characteristics necessary for training a generalizing segmentation network.
• Figure 1: Virtual Scene Structure used in DiffRenderGAN Experiments. Here, for visualization, emissions are omitted, and the stage and rectangle light source meshes are scaled in the side view. The stage and nanoparticle meshes are enclosed within a toroidal mesh that acts as a light source, creating glowing edge effects similar to those seen in ion or electron microscopy.
• Figure 2: DiffRenderGAN Training Procedure. Domain experts create target nanomaterial meshes to match the morphology of real particle systems. Scale and placement parameters are used to compute a transformation matrix for training. The meshes and transformation matrix serve as input to the DiffRenderGAN model. During image generation, a slice of the matrix is processed by a 5-layer Fully Connected Network (FCN) to predict BSDF parameters and noise scale. These parameters are passed to a differentiable renderer, which uses a virtual scene with scaled and positioned meshes to create the final synthetic nanomaterial image. A technical description of DiffRenderGAN's modules is provided in Section \ref{['m_model_design']}. For visualization purposes, the virtual scene used by the differentiable renderer is shown in a simplified form. The actual structure can be found in the supplementary information of this paper, with a technical summary stated in Section \ref{['m_virtual_scene']}.
• Figure 2: DiffRenderGAN Training FID Score Progression Over 50 Training Epochs. This figure shows the FID scores for the four experiments discussed in the main text. Lower FID values indicate greater dataset similarity between generated and real images.
• Figure 3: Comparison of Real and Synthetic Image Patches with Corresponding Segmentation Masks. In each figure section, the top row shows real images used to train DiffRenderGAN, the middle row depicts synthetic images, and the bottom row shows the corresponding segmentation masks, highlighting material classes (purple) and boundaries (orange). These synthetic image-mask pairs serve as training data for multiclass segmentation networks as demonstrated in Section \ref{['Evaluation']}. a. AgNW: trained using 10 bent cone meshes, choosing for transformation computation random placement in 2D and a lognormal size distribution. b. TiO$_2$ in SEM from Rühle et al. (2021) ruhle2021workflow: trained using 40 cubically deformed meshes, choosing for transformation computation Poisson Disk-based placement in 3D and a lognormal size distribution. c. TiO$_2$ in HIM from Mill et al. (2021) mill2021synthetic: trained using 15 cubically deformed meshes, choosing for transformation computation Poisson Disk-based placement in 3D and a lognormal size distribution. d. SiO$_2$ in HIM from Mill et al. (2021) mill2021synthetic: trained using 20 sphere meshes, choosing for transformation computation Poisson Disk-based placement in 3D and a lognormal size distribution.