CuNeRF: Cube-Based Neural Radiance Field for Zero-Shot Medical Image Arbitrary-Scale Super Resolution

TL;DR

CuNeRF tackles medical image arbitrary-scale super-resolution without HR references by learning a continuous coordinate-intensity representation from a single LR volume. It introduces cube-based sampling, isotropic volume rendering, and cube-based hierarchical rendering to address hole artifacts inherent to NeRF in medical data, enabling free-viewpoint and arbitrary-scale slice synthesis. The approach yields state-of-the-art or competitive results on MRI and CT across 3D MISR and volumetric MISR tasks, with robust ablations affirming each component's contribution. By eliminating the need for LR-HR training pairs, CuNeRF offers a flexible, zero-shot solution with practical clinical utility, and code is publicly released.

Abstract

Medical image arbitrary-scale super-resolution (MIASSR) has recently gained widespread attention, aiming to super sample medical volumes at arbitrary scales via a single model. However, existing MIASSR methods face two major limitations: (i) reliance on high-resolution (HR) volumes and (ii) limited generalization ability, which restricts their application in various scenarios. To overcome these limitations, we propose Cube-based Neural Radiance Field (CuNeRF), a zero-shot MIASSR framework that can yield medical images at arbitrary scales and viewpoints in a continuous domain. Unlike existing MIASSR methods that fit the mapping between low-resolution (LR) and HR volumes, CuNeRF focuses on building a coordinate-intensity continuous representation from LR volumes without the need for HR references. This is achieved by the proposed differentiable modules: including cube-based sampling, isotropic volume rendering, and cube-based hierarchical rendering. Through extensive experiments on magnetic resource imaging (MRI) and computed tomography (CT) modalities, we demonstrate that CuNeRF outperforms state-of-the-art MIASSR methods. CuNeRF yields better visual verisimilitude and reduces aliasing artifacts at various upsampling factors. Moreover, our CuNeRF does not need any LR-HR training pairs, which is more flexible and easier to be used than others. Our code is released at https://github.com/NarcissusEx/CuNeRF.

Figures (8)

• Figure 1: CuNeRF is the first zero-shot Medical Image Arbitrary-Scale Super Resolution framework. After training on an LR medical volume (a) itself, CuNeRF can build the corresponding continuous volumetric representation (b), which is able to achieve (c) Free-viewpoint slice synthesis: yielding novel-view medical slices from the arbitrary viewpoints, and (d) Arbitrary-scale super-resolution: upsampling medical images at arbitrary scales in a continuous domain. Project page is available at https://narcissusex.github.io/CuNeRF.
• Figure 2: Difference between existing supervised MISR (a), zero-shot MISR (ZSMISR) (b) and CuNeRF(c). Visually, supervised MISR methods need to collect considerable LR-HR pairs for training, while ZSMISR and our CuNeRF only train the model on each test volume itself. However, given a test volume, ZSMISR methods can only upsample medical images at a specific scale (one-for-one), while our CuNeRF can handle arbitrary upsampling scales (one-for-all).
• Figure 3: Visual examples of 3D MISR at $\times$2.5 factor between ArSSR ARSSR, NeRF$^\dag$NeRF and our CuNeRF on MSD MSD dataset. Heatmaps at the bottom visualize the difference between the results and the HR image. Visually, NeRF$^\dag$ yields grid-like artifacts, and ArSSR produces non-existent details. By contrast, our CuNeRF achieves better visual verisimilitude and fewer artifacts.
• Figure 4: Visualization of the sampling strategies between NeRF NeRF(a) and CuNeRF(b) applied on medical volumes. Visually, NeRF only samples the rays corresponding to each training pixel, which cannot cover the whole representation fields, leaving some "holes" (i.e., unmodeled space within between adjacent training pixels. To address this issue, CuNeRF samples cubes centered by each training pixel, and therefore the "holes" are well-covered by the spatial overlaps.
• Figure 5: The overall framework of our CuNeRF. To synthesize a pixel (red circle) with the spatial position $\mathbf{x}_t$, (a)CuNeRF first uniformly samples $N$ points as a point set $\{\hat{\mathbf{x}}_i\}^N_{i=1}$ within the cube space (purple cube) centered by $\mathbf{x}_t$. Then, CuNeRF obtains the coarse estimation (blue cube) by feeding the sampling points into an MLP $F_{\Theta}$ to produce the set of corresponding pixel intensity $\{c_i\}_{i=1}^N$ and volume density $\{\sigma_i\}_{i=1}^N$. (b) Subsequently, assuming $\sigma$ of each sampling point is only related to the distance with the cube center $\mathbf{x}_t$, CuNeRF computes the coarse output of the target pixel via volume integral. (c) Finally, CuNeRF resamples the points under the probability density function (PDF) of coarse estimation to acquire the fine estimation (orange cube) of the cube. The fine output is generated by the same procedures as (b). Since these two rendering functions are differentiable, CuNeRF can be optimized by minimizing the rendering loss in Eq \ref{['eq:render_loss']}. The fine output is the final rendering result of the target spatial position $\mathbf{x}_t$.