Boosting 3D Liver Shape Datasets with Diffusion Models and Implicit Neural Representations

TL;DR

The paper tackles the scarcity and artifacts in open 3D liver shape datasets by introducing an unconditional 3D liver HyperDiffusion framework that operates on implicit neural representations (INRs) and HyperNetworks. It presents a two-stage pipeline: first learn per-object occupancy MLPs to faithfully encode liver surfaces, then model the space of those MLP weights with a Transformer-based diffusion model to generate new, plausible liver shapes, reconstructed via Marching Cubes. Quantitative metrics and expert evaluation indicate that the synthesized INRs closely resemble real liver morphology and can augment data quality for 3D reconstruction tasks, while outperforming a 1D UNet baseline in several metrics. The work suggests broad applicability to 3D medical imaging and outlines future directions toward conditional generation (e.g., text-to-3D, image-to-3D) to further control synthesized anatomy.

Abstract

While the availability of open 3D medical shape datasets is increasing, offering substantial benefits to the research community, we have found that many of these datasets are, unfortunately, disorganized and contain artifacts. These issues limit the development and training of robust models, particularly for accurate 3D reconstruction tasks. In this paper, we examine the current state of available 3D liver shape datasets and propose a solution using diffusion models combined with implicit neural representations (INRs) to augment and expand existing datasets. Our approach utilizes the generative capabilities of diffusion models to create realistic, diverse 3D liver shapes, capturing a wide range of anatomical variations and addressing the problem of data scarcity. Experimental results indicate that our method enhances dataset diversity, providing a scalable solution to improve the accuracy and reliability of 3D liver reconstruction and generation in medical applications. Finally, we suggest that diffusion models can also be applied to other downstream tasks in 3D medical imaging.

Figures (5)

• Figure 1: Our analysis of TotalSegmentator, MedShapeNet, and SARAMIS shows that current 3D medical shape datasets are largely built on 3D segmentation results, such as those from TotalSegmentator. This approach results in disorganized datasets, with usable liver objects (labeled 'Usable' as a green bar) accounting for only $48.14\%$, while the others (labeled 'No full shape', 'Not usable', 'Not sure', and 'Requires editing' as red bars) are prone to artifacts and largely unusable in the case of TotalSegmentator.
• Figure 2: Overview of the proposed 3D liver HyperDiffusion framework. Our framework consists of two training stages: (a) MLP training and (b) HyperDiffusion on MLP training. (c) To synthesize a novel 3D liver object, we use the optimized HyperDiffusion network to sample the novel $\theta$ from the diffusion process. The novel $\theta$ is reshaped back to an MLP and used to reconstruct the novel 3D liver object through the Marching Cubes (MC) algorithm. (d) Visualization of the novel 3D liver objects.
• Figure 3: Our workflow used for MLP training. From the 3D liver object, we sample a point cloud both inside and outside the object's surface. Given the coordinates of each point as input, the MLP predicts the occupancy value: $1$ for inside (red point color) and $0$ for outside (gray point color). To reconstruct the 3D liver object from the volume of occupancy values, we use the MC algorithm.
• Figure 4: The operation in HyperDiffusion during training. The flattened $\theta$ is added random Gaussian noise and split into a sequence of tokens through projection. An additional time step token is appended to the sequence to indicate the current time step $t$, and the sequence is then added to a learnable Positional Encoding (PE) before being input into the Transformer. The output tokens are projected back to the size of the weights and biases to merge and represent the denoised weights $\hat{\theta}^{(t)}$.
• Figure 5: (a) The GUI allows the user to view one 3D liver object at a time and navigate among 150 objects to review and select from the options -- Real, Fake, and Not sure -- to classify the current 3D liver object. (b) Visualization of two synthesized 3D liver objects for each type of classification.