Neural Radiance Fields in Medical Imaging: A Survey

TL;DR

This survey addresses the problem of applying Neural Radiance Fields (NeRFs) to medical imaging, where modality-specific physics and the need for detailed inner-structure visualization pose unique challenges. It surveys organ-specific NeRF methods (e.g., MedNeRF, UMedNeRF, Masked-NeRF, NAF, SNAF), data generation via Digitally Reconstructed Radiographs (DRR), and evaluation protocols, while outlining current limitations and future directions. The paper contributes a organ-based taxonomy of NeRF approaches, discusses dataset resources, and presents evaluation metrics to benchmark 2D rendering and 3D projections. The work highlights potential clinical impact, including reduced radiation exposure, faster image synthesis, and enhanced diagnostic and surgical planning capabilities, while calling for advances in resolution, boundary delineation, color-density modeling, and computational efficiency.

Abstract

Neural Radiance Fields (NeRF), as a pioneering technique in computer vision, offer great potential to revolutionize medical imaging by synthesizing three-dimensional representations from the projected two-dimensional image data. However, they face unique challenges when applied to medical applications. This paper presents a comprehensive examination of applications of NeRFs in medical imaging, highlighting four imminent challenges, including fundamental imaging principles, inner structure requirement, object boundary definition, and color density significance. We discuss current methods on different organs and discuss related limitations. We also review several datasets and evaluation metrics and propose several promising directions for future research.

Figures (5)

• Figure 1: Overview of this work, discuss NeRF for medical imaging of different organs. Top: NeRF model mildenhall2021nerf. Bottom: An application of NeRF in medical images, the goal is to render the 3D volume from a single view corona2022mednerfhu2023umednerf.
• Figure 2: Overview of the challenges of NeRFs in medical images. Top: The fundamental imaging principles of RBG image and medical image are different, an example of X-rays imaging is from cai2023structure. Bottom: (left) Medical images necessitate detailed inner structure visualization. (mid) often suffer from poor object boundary definition, and (right) exhibit the varying significance of color density.
• Figure 3: Over view of Uncertainty-aware MedNeRF hu2023umednerf. Top: Network structures and training phrase corona2022mednerf. Mid: Fine-tuning the generator for balancing the blurriness and accuracy with uncertainty-aware multi-task loss. Bottom: Rendering the whole 3D volume (each view taken at a 5-degree interval) using a single view slide.
• Figure 4: (a) Overview of extended-MedNeRF with 2D Slices of Brain MRI scans in iddrisu20233d. (b) Overview of Masked NeRF zhou2023robust. Left: Skull projections from NeRF. Right. Skull projections from Masked NeRF scene representations. (c) Overview of Coronary Angiography WikiCoronaryCath2023. (d) Overview of Sparse-view Neural Attenuation Fields (SNAF) fang2022snaf. An X-ray source systematically traverses around the region of interest, producing sparse projections captured by a flat-panel detector positioned on the opposite side.
• Figure 5: The relation between Digitally Reconstructed Radiograph (DRR) and medical imaging NeRFs, the DRR example is from montufar2018perspective. Left: leverages original CT scan slices, Mid: perceived as a three-dimensional voxel array, Right: to generate a DRR. This direct volume rendering scheme efficiently transforms the volumetric data from a CT scan into a visually comprehensive DRR, offering a synthesized view similar to traditional X-ray imagery.