Synthesis of Annotated Colorectal Cancer Tissue Images from Gland Layout

TL;DR

The paper tackles the scarcity of richly annotated histopathology data by introducing an interactive framework that jointly synthesizes colorectal tissue images and gland masks from gland layouts. It combines a per-gland latent embedding and a mask generator to produce glandular masks, which are wrapped into a tissue mask and passed to a Pix2Pix-like encoder–decoder to generate tissue images, supervised by three discriminators. A key novelty is the latent-diffusion-based synthesis of glandular masks via a VQ-VAE, enabling mask generation without fixed layouts and conditioning on cancer type. Quantitative results show competitive FID scores and strong gland-segmentation validation on synthetic data, demonstrating potential for scalable generation of annotated histology pairs and downstream evaluation tasks.

Abstract

Generating realistic tissue images with annotations is a challenging task that is important in many computational histopathology applications. Synthetically generated images and annotations are valuable for training and evaluating algorithms in this domain. To address this, we propose an interactive framework generating pairs of realistic colorectal cancer histology images with corresponding glandular masks from glandular structure layouts. The framework accurately captures vital features like stroma, goblet cells, and glandular lumen. Users can control gland appearance by adjusting parameters such as the number of glands, their locations, and sizes. The generated images exhibit good Frechet Inception Distance (FID) scores compared to the state-of-the-art image-to-image translation model. Additionally, we demonstrate the utility of our synthetic annotations for evaluating gland segmentation algorithms. Furthermore, we present a methodology for constructing glandular masks using advanced deep generative models, such as latent diffusion models. These masks enable tissue image generation through a residual encoder-decoder network.

Figures (5)

• Figure 1: Block diagram of the proposed framework: (a) shows an input gland layout where glands are arranged on 2-d spatial locations. Each gland is characterized by a gland specific vector which undergoes affine transformation to form latent vectors (b). Each latent vector is consumed by the individual mask generator which outputs binary individual glandular masks (c). The generated individual masks along with input bounding boxes act as an input to the bilinear interpolation algorithm, which wraps generated masks inside bounding boxes creating the intermediate tensor (d). The mask generator network consumes the intermediate tensor and generates the glandular mask (e), which is then passed through the encoder-decoder generator network generating the final tissue image (f). There are three discriminators employed for generated mask, image and glandular parts inside the image.
• Figure 2: Visual results of generated colorectal tissue images along with their gland segmentation masks from input gland layouts (a). (b) shows original gland segmentation masks while (c) shows the ground truth tissue images.
• Figure 3: The leftmost image (a) shows the generated sample out from the proposed framework. Images on the right to it shows the change in appearance of the yellow bordered gland, after altering location $\overrightarrow{l}=(l_x,l_y)$ and size $\overrightarrow{s}=(s_x,s_y)$. (b) and (c) shows the shift of that gland to left side (lowering $l_x$) and upwards (increasing $l_y$), respectively. For the same gland, (d) shows the contraction horizontally and (e) shows expansion vertically after reducing ($s_x$) and increasing ($s_y$), respectively.
• Figure 4: Samples of both real (above) and constructed (below) annotated pairs of tissue images and corresponding gland segmentation masks. The masks shown on the right side are generated from the U-net based segmentation algorithm when applied on original (above) and synthetic (below) images.
• Figure 5: Annotated pairs synthesized using the latent diffusion model. The first two pairs (from right) depict annotated colon tissue images of the benign grade, while the last two show images of the malignant type.