Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Intraoperative Registration by Cross-Modal Inverse Neural Rendering

Maximilian Fehrentz, Mohammad Farid Azampour, Reuben Dorent, Hassan Rasheed, Colin Galvin, Alexandra Golby, William M. Wells, Sarah Frisken, Nassir Navab, Nazim Haouchine

TL;DR

This work tackles 3D/2D intraoperative registration in neurosurgery by framing it as a differentiable, cross-modal rendering problem. It introduces a neural radiance field that disentangles preoperative anatomy (structure) from intraoperative appearance (style) via a multi-style hypernetwork conditioned on the intraoperative image, enabling pose estimation through gradient-based optimization of a rendering loss. Training leverages neural style transfer to create style-diverse appearances while preserving anatomy, and inference relies on a differentiable renderer to compute pose updates in SE($3$). Empirical results on synthetic targets and retrospective clinical cases show improved accuracy over baselines, meeting clinical standards in most styles, with qualitative success in real cases and a clear path toward handling non-rigid deformations in future work.

Abstract

We present in this paper a novel approach for 3D/2D intraoperative registration during neurosurgery via cross-modal inverse neural rendering. Our approach separates implicit neural representation into two components, handling anatomical structure preoperatively and appearance intraoperatively. This disentanglement is achieved by controlling a Neural Radiance Field's appearance with a multi-style hypernetwork. Once trained, the implicit neural representation serves as a differentiable rendering engine, which can be used to estimate the surgical camera pose by minimizing the dissimilarity between its rendered images and the target intraoperative image. We tested our method on retrospective patients' data from clinical cases, showing that our method outperforms state-of-the-art while meeting current clinical standards for registration. Code and additional resources can be found at https://maxfehrentz.github.io/style-ngp/.

Intraoperative Registration by Cross-Modal Inverse Neural Rendering

TL;DR

This work tackles 3D/2D intraoperative registration in neurosurgery by framing it as a differentiable, cross-modal rendering problem. It introduces a neural radiance field that disentangles preoperative anatomy (structure) from intraoperative appearance (style) via a multi-style hypernetwork conditioned on the intraoperative image, enabling pose estimation through gradient-based optimization of a rendering loss. Training leverages neural style transfer to create style-diverse appearances while preserving anatomy, and inference relies on a differentiable renderer to compute pose updates in SE(). Empirical results on synthetic targets and retrospective clinical cases show improved accuracy over baselines, meeting clinical standards in most styles, with qualitative success in real cases and a clear path toward handling non-rigid deformations in future work.

Abstract

We present in this paper a novel approach for 3D/2D intraoperative registration during neurosurgery via cross-modal inverse neural rendering. Our approach separates implicit neural representation into two components, handling anatomical structure preoperatively and appearance intraoperatively. This disentanglement is achieved by controlling a Neural Radiance Field's appearance with a multi-style hypernetwork. Once trained, the implicit neural representation serves as a differentiable rendering engine, which can be used to estimate the surgical camera pose by minimizing the dissimilarity between its rendered images and the target intraoperative image. We tested our method on retrospective patients' data from clinical cases, showing that our method outperforms state-of-the-art while meeting current clinical standards for registration. Code and additional resources can be found at https://maxfehrentz.github.io/style-ngp/.
Paper Structure (15 sections, 6 equations, 4 figures, 1 table)

This paper contains 15 sections, 6 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contribution
  3. Methods
  4. Problem Formulation & Overview
  5. Bridging the Domain Gap via Hypernetwork Multi-Style NeRF
  6. NeRF Background.
  7. Training the Hypernetwork Multi-Style NeRF.
  8. Intraoperative Registration
  9. Experiments and Results
  10. Datasets.
  11. View Synthesis.
  12. Pose Estimation on Synthetic Targets.
  13. Comparison with Baseline and State-of-the-art.
  14. Tests on Clinical Cases.
  15. Conclusion

Figures (4)

  • Figure 1: Left (preoperative): We use a NeRF to first learn density/anatomy (orange) from a mesh $\mathbf{M}$ extracted from an MR scan, then learn single-shot style adaptation (blue) through a hypernetwork $h$ while freezing the rest of the NeRF, keeping the previously learned density $f_d$ fixed. Right (intraoperative): Iterative pose estimation on target $\mathbf{I}$. The trained NeRF and hypernet (green highlights) are used as style-conditioned neural rendering engine using ray marching, with $f$ adapted to the appearance of the intraoperative registration target $\mathbf{I}$ through the hypernetwork $h$.
  • Figure 2: An example of synthesis from 3 different poses on one of the clinical cases. First: image obtained from the MRI with surgical microscope image $\mathbf{I}$. Remaining images: synthesis with $f$, style inferred by hypernetwork $h$ on $\mathbf{I}$.
  • Figure 3: Evaluation on synthetic targets (from left to right): cross-correlation matrix of Gram-similarity score of all styles showing pairwise style similarity and dissimilarity; pose distribution (blue: training set, red: test set); and accuracy-threshold curves for rotation and translation.
  • Figure 4: Tests on real cases, one case per row. From left to right: preoperative MR scan of the area (volume rendering), intraoperative target image from the surgical microscope, 3 optimization steps (early-optimization, mid-optimization, and final pose estimation), and intraoperative image with vessel-overlay of our estimated pose.