SG-GAN: Fine Stereoscopic-Aware Generation for 3D Brain Point Cloud Up-sampling from a Single Image

TL;DR

SG-GAN addresses the challenge of producing high-density 3D brain point clouds from a single MRI for minimally invasive neurosurgery. It combines a parameter-free Free Transforming Module encoder with two stage-guided, graph-based GANs: a Stage-I tree-structured GCN to outline the brain contour and a Stage-II stack-structured GCN to upsample and refine details. The framework is trained with adversarial losses and geometry-aware terms to enforce global consistency and local fidelity, and is validated on a brain MRI dataset against state-of-the-art upsampling methods, showing improvements in CD, EMD, HD, and PC-to-PC error, as well as higher realism in a classification task. This approach offers a practical, fast route to real-time, high-density 3D brain reconstructions that can enhance intraoperative navigation and planning. Future work includes clinical data collection to broaden applicability and further reduce input constraints.

Abstract

In minimally-invasive brain surgeries with indirect and narrow operating environments, 3D brain reconstruction is crucial. However, as requirements of accuracy for some new minimally-invasive surgeries (such as brain-computer interface surgery) are higher and higher, the outputs of conventional 3D reconstruction, such as point cloud (PC), are facing the challenges that sample points are too sparse and the precision is insufficient. On the other hand, there is a scarcity of high-density point cloud datasets, which makes it challenging to train models for direct reconstruction of high-density brain point clouds. In this work, a novel model named stereoscopic-aware graph generative adversarial network (SG-GAN) with two stages is proposed to generate fine high-density PC conditioned on a single image. The Stage-I GAN sketches the primitive shape and basic structure of the organ based on the given image, yielding Stage-I point clouds. The Stage-II GAN takes the results from Stage-I and generates high-density point clouds with detailed features. The Stage-II GAN is capable of correcting defects and restoring the detailed features of the region of interest (ROI) through the up-sampling process. Furthermore, a parameter-free-attention-based free-transforming module is developed to learn the efficient features of input, while upholding a promising performance. Comparing with the existing methods, the SG-GAN model shows superior performance in terms of visual quality, objective measurements, and performance in classification, as demonstrated by comprehensive results measured by several evaluation metrics including PC-to-PC error and Chamfer distance.

Figures (11)

• Figure 1: Illustration of the SG-GAN. The encoder, composed of the free transforming module, is highlighted with pink. The stage-I GAN is marked with blue, while the stage-II GAN is marked with green. More details of several core blocks are given in the following sub-sections
• Figure 2: Comparison of a) the traditional self-attention mechanism and b) the proposed FTM.
• Figure 3: Illustrations for the stage-I and stage-II generators. The stage-I generator based on the multi-geometry graph convolutional neural network describe the branch structure of point features and outline the fundamental outline of the target. The stage-II generator acquire the results from Stage-I and generates high-density point clouds with detailed features and it is capable of correcting defects and restoring the detailed features of the region of interest through the up-sampling process.
• Figure 4: The comparison of microstructure in the same generating of the stage-I GAN and stage-II GAN.
• Figure 5: Reconstructed colored samples of 3D brain measured by PC-to-PC error. Heat map is used to show the exact value of the error.