Recover Biological Structure from Sparse-View Diffraction Images with Neural Volumetric Prior

TL;DR

This work tackles the challenge of reconstructing 3D refractive-index distributions from sparse-view diffraction images in microscopy. It introduces Neural Volumetric Prior (NVP), a hybrid explicit-implicit neural representation that encodes volumetric structure in an adaptively sampled grid and refines it with an MLP, all under a physics-informed diffraction rendering model. By integrating coherent alignment and self-calibration, NVP achieves high-fidelity RI reconstructions with substantially fewer measurements (≈50x fewer views) and faster processing (≈3x faster) than prior diffraction tomography methods, demonstrated on synthetic and real biological data, including MDCK cells. The approach enables real-time, 3D imaging of dynamic biological processes and lays groundwork for extending to other diffraction-based modalities and priors, with potential integration of 3D convolutions or diffusion-based priors to further boost performance.

Abstract

Volumetric reconstruction of label-free living cells from non-destructive optical microscopic images reveals cellular metabolism in native environments. However, current optical tomography techniques require hundreds of 2D images to reconstruct a 3D volume, hindering them from intravital imaging of biological samples undergoing rapid dynamics. This poses the challenge of reconstructing the entire volume of semi-transparent biological samples from sparse views due to the restricted viewing angles of microscopes and the limited number of measurements. In this work, we develop Neural Volumetric Prior (NVP) for high-fidelity volumetric reconstruction of semi-transparent biological samples from sparse-view microscopic images. NVP integrates explicit and implicit neural representations and incorporates the physical prior of diffractive optics. We validate NVP on both simulated data and experimentally captured microscopic images. Compared to previous methods, NVP significantly reduces the required number of images by nearly 50-fold and processing time by 3-fold while maintaining state-of-the-art performance. NVP is the first technique to enable volumetric reconstruction of label-free biological samples from sparse-view microscopic images, paving the way for real-time 3D imaging of dynamically changing biological samples. \href{https://xue-lab-cobi.github.io/Sparse-View-FDT/}{Project Page}

Figures (9)

• Figure 1: Illustrations of different neural representation methods. (a) Explicit neural representation: A non-parametric method in which the refractive index (RI) distribution $\hat{n}$ is directly reconstructed by the projection function $W_{\text{enr}}(x, y, z)$, which provide a one-to-one mapping from spatial coordinates to RI values Xue:22FDT. (b) Implicit neural representation: Instead of directly reconstructing $\hat{n}$, this method optimizes the parameters of a multi-layer perceptron (MLP) $F_{\text{inr}}$, which is then used to predict $\hat{n}$decaf. (c) Triplane: A hybrid method combining non-parametric and parametric components by solving both the triplane features ${W_{xy}, W_{xz}, W_{yz}}$ and the neural model $F_{\text{tri}}$ to reconstruct $\hat{n}$kplanes_2023. (d) NVP: Our proposed hybrid approach reconstructs $\hat{n}$ by integrating an uncompressed volumetric prior ${W_{xyz}}$ into the neural model $F_{\text{nvp}}$.
• Figure 2: Overview of our method for 3D reconstruction. a. Neural volumetric prior (NVP): Predefined 3D grids are reshaped and processed by MLPs to generate the predicted 3D RI volume $\hat{n}$. b. Multi-slice rendering equation: The multi-slice model calculates light propagation through the volumetric sample from the fluorescence sources (white spots at the bottom) by accounting for light diffraction at each slice. Each illumination configuration of the fluorescence sources (e.g., $F_1, F_i, F_n$) interacts with the volume to produce a corresponding set of rendered images ($\hat{I}_1, \hat{I}_i, \hat{I}_n$). The illumination configurations are jointly optimized with the RI volume. c. Loss functions: The predicted images $\hat{I}_{\text{pred}}$ are aligned with ground truth images $I_{\text{GT}}$ and masked based on light coherence, resulting in $\hat{I}_{\text{mask}}$ and $I_{\text{mask}}$. Loss functions, including $L1$, $L2$, and SSIM, are calculated over the masked regions of $\hat{I}_{\text{pred}}$ and $I_{\text{GT}}$, and a total variation (TV) regularizer $R(\hat{n})$ is applied.
• Figure 3: NVP architecture. Adaptive feature grids $(H, W, L, F)$ are first reshaped into feature vectors of size $(N, F)$, where $H, W, L$ are the volume dimensions and $F$ is the feature dimension. These vectors are then passed through MLP layers to predict the RI of each voxel.
• Figure 4: NVP reconstructs the 3D RI of the synthetic cell dataset with micron-scale resolution and excellent optical sectioning. (a) Reconstructed 3D RI distribution showing the spatial distribution of cells within a 154 $\mu$m $\times$ 154 $\mu$m $\times$ 24 $\mu$m volume. (b) Predicted fluorescence images under the illumination from two representative fluorescence sources, demonstrating the accuracy of the reconstructed model compared to the ground-truth data. (c) Cross-sectional comparison of RI reconstructed by NVP (top) and ground-truth (bottom) along the axial direction, illustrating consistency between the reconstructed and ground-truth RI distribution across the sample volume. Color bars indicate intensity ranges for predicted images, absolute errors, and RI values.
• Figure 5: NVP achieves more accurate reconstruction compared to the explicit and triplane methods. The top row shows the RI maps produced by each method, with the ground-truth RI map included for reference. The bottom row displays the corresponding error maps, highlighting the reconstruction errors in each method. NVP demonstrates superior performance, producing smoother reconstructions and lower errors that closely match the ground truth.