An Endoscopic Chisel: Intraoperative Imaging Carves 3D Anatomical Models

TL;DR

The paper tackles the problem of navigation inaccuracies during functional endoscopic sinus surgery caused by intraoperative anatomical changes not captured in preoperative CT. It introduces a vision-based pipeline that builds and sequentially updates a TSDF-based preoperative sinus model using intraoperative endoscopic video and known camera poses, guided by monocular depth estimates and a change-detection step. Key contributions include the first vision-based intraoperative model updating method for sinus surgery, a TSDF fusion approach that selectively updates changed regions, and ex vivo validation showing reduced depth errors with updates plus ablation analyses. This work advances toward a digital-twin paradigm and endoscope-centered navigation in sinus surgery by enabling continuous, image-driven refinement of the patient model during procedures.

Abstract

Purpose: Preoperative imaging plays a pivotal role in sinus surgery where CTs offer patient-specific insights of complex anatomy, enabling real-time intraoperative navigation to complement endoscopy imaging. However, surgery elicits anatomical changes not represented in the preoperative model, generating an inaccurate basis for navigation during surgery progression. Methods: We propose a first vision-based approach to update the preoperative 3D anatomical model leveraging intraoperative endoscopic video for navigated sinus surgery where relative camera poses are known. We rely on comparisons of intraoperative monocular depth estimates and preoperative depth renders to identify modified regions. The new depths are integrated in these regions through volumetric fusion in a truncated signed distance function representation to generate an intraoperative 3D model that reflects tissue manipulation. Results: We quantitatively evaluate our approach by sequentially updating models for a five-step surgical progression in an ex vivo specimen. We compute the error between correspondences from the updated model and ground-truth intraoperative CT in the region of anatomical modification. The resulting models show a decrease in error during surgical progression as opposed to increasing when no update is employed. Conclusion: Our findings suggest that preoperative 3D anatomical models can be updated using intraoperative endoscopy video in navigated sinus surgery. Future work will investigate improvements to monocular depth estimation as well as removing the need for external navigation systems. The resulting ability to continuously update the patient model may provide surgeons with a more precise understanding of the current anatomical state and paves the way toward a digital twin paradigm for sinus surgery.

Figures (6)

• Figure 1: Intraoperative camera positions as recorded via optical tracking in the preoperative CT, shown in tissues that were removed during surgery. The pink points show the endoscope path through the nasal cavity, while the green point is where the path violates the CT information, as it is in a region where there previously was tissue but is now vacant for the endoscope to pass through.
• Figure 2: Overview of method for intraoperative update. Note that the preoperative model is used at surgical step $t = 0$ and depths are rendered from the previous model ($D_(\mathbf{X})_t$) using the camera pose $i$ from the current intraoperative sequence (i.e. $T_{t+1}^i$)
• Figure 3: Framewise change detection.
• Figure 4: Endoscopic video frames depicting anatomical change with corresponding depths from preoperative and intraoperative CTs, estimation, and rendered from the update and depth ablation meshes. The regions modified between each intraoperative step are highlighted in red. Note that the depths are rendered from the associated camera pose of the frame, and these poses differ between sequences.
• Figure 5: Truncated signed distance function representation using a discrete voxel grid of the CT. We employ the cameras and corresponding depth maps to extract the distance from each voxel to the anatomical surface.