MimickNet, Matching Clinical Post-Processing Under Realistic Black-Box Constraints

TL;DR

This is the first work to approximate current clinical-grade ultrasound post-processing under realistic black-box constraints where before and after post-processing data is unavailable and the MimickNet software is made open source.

Abstract

Image post-processing is used in clinical-grade ultrasound scanners to improve image quality (e.g., reduce speckle noise and enhance contrast). These post-processing techniques vary across manufacturers and are generally kept proprietary, which presents a challenge for researchers looking to match current clinical-grade workflows. We introduce a deep learning framework, MimickNet, that transforms raw conventional delay-and-summed (DAS) beams into the approximate post-processed images found on clinical-grade scanners. Training MimickNet only requires post-processed image samples from a scanner of interest without the need for explicit pairing to raw DAS data. This flexibility allows it to hypothetically approximate any manufacturer's post-processing without access to the pre-processed data. MimickNet generates images with an average similarity index measurement (SSIM) of 0.930$\pm$0.0892 on a 300 cineloop test set, and it generalizes to cardiac cineloops outside of our train-test distribution achieving an SSIM of 0.967$\pm$0.002. We also explore the theoretical SSIM achievable by evaluating MimickNet performance when trained under gray-box constraints (i.e., when both pre-processed and post-processed images are available). To our knowledge, this is the first work to establish deep learning models that closely approximate current clinical-grade ultrasound post-processing under realistic black-box constraints where before and after post-processing data is unavailable. MimickNet serves as a clinical post-processing baseline for future works in ultrasound image formation to compare against. To this end, we have made the MimickNet software open source.

Figures (6)

• Figure 1: Fetal image comparing clinical-grade post-processed images (ground truth) and MimickNet post-processing. In the last row, the difference between clinical-grade and MimickNet post-processing is scaled to maximize dynamic range. The SSIM of the MimickNet image to clinical grade image is 0.972 and the PSNR is 26.78.
• Figure 2: Above is a diagram of the generator and discriminator structure for MimickNet in one translation direction. Note: the reverse translation direction uses an identical mirrored structure. Under gray-box training constraints, only the generator is used.
• Figure 3: Liver (left) and phantom (right) images. The difference between clinical-grade and MimickNet outputs are scaled to maximize dynamic range. The SSIM and PSNR between MimickNet and clinical-grade images for the liver target is 0.9472 and 26.91, respectively. The SSIM and PSNR between MimickNet and clinical-grade images for the phantom target is 0.9802 and 27.20, respectively.
• Figure 4: The distribution of contrast-structure (top) and luminance (bottom) of all image frames in our test dataset produced under gray-box and black-box constraints. The $cs$ is $0.987 \pm 0.005$ and $l$ is $0.993 \pm 0.0103$ under gray-box constraints. The $cs$ is $0.978 \pm 0.008$ and $l$ is $0.967 \pm 0.073$ under black-box constraints.
• Figure 5: The worst case scenario images for two fetal brain images (top, bottom), and a phantom (middle). The SSIM of the black-box case (MimickNet) to the ground truth images from top to bottom is 0.665 ($cs$ = 0.962, $l$=0.681), 0.414 ($cs$ = 0.947, $l$ = 0.419), and 0.603 ($cs$ = 0.964, $l$ = 0.612). The SSIM of the gray-box case to the ground truth images from top to bottom is 0.873 ($cs$=0.984, $l$=0.883), 0.967 ($cs$=0.996, $l$=0.971), and 0.901 ($cs$=0.988, $l$=0.911). Here $l$ is the luminance and $cs$ is the contrast-structure components of SSIM.