Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FoundationPose-Initialized 3D-2D Liver Registration for Surgical Augmented Reality

Hanyuan Zhang, Lucas He, Runlong He, Abdolrahim Kadkhodamohammadi, Danail Stoyanov, Brian R. Davidson, Evangelos B. Mazomenos, Matthew J. Clarkson

TL;DR

Combined rigid-NICP registration outperformed rigid-only registration, demonstrating NICP as an efficient substitute for finite-element deformable models and offering a lightweight, engineering-friendly alternative to FE-based deformation.

Abstract

Augmented reality can improve tumor localization in laparoscopic liver surgery. Existing registration pipelines typically depend on organ contours; deformable (non-rigid) alignment is often handled with finite-element (FE) models coupled to dimensionality-reduction or machine-learning components. We integrate laparoscopic depth maps with a foundation pose estimator for camera-liver pose estimation and replace FE-based deformation with non-rigid iterative closest point (NICP) to lower engineering/modeling complexity and expertise requirements. On real patient data, the depth-augmented foundation pose approach achieved 9.91 mm mean registration error in 3 cases. Combined rigid-NICP registration outperformed rigid-only registration, demonstrating NICP as an efficient substitute for finite-element deformable models. This pipeline achieves clinically relevant accuracy while offering a lightweight, engineering-friendly alternative to FE-based deformation.

FoundationPose-Initialized 3D-2D Liver Registration for Surgical Augmented Reality

TL;DR

Combined rigid-NICP registration outperformed rigid-only registration, demonstrating NICP as an efficient substitute for finite-element deformable models and offering a lightweight, engineering-friendly alternative to FE-based deformation.

Abstract

Augmented reality can improve tumor localization in laparoscopic liver surgery. Existing registration pipelines typically depend on organ contours; deformable (non-rigid) alignment is often handled with finite-element (FE) models coupled to dimensionality-reduction or machine-learning components. We integrate laparoscopic depth maps with a foundation pose estimator for camera-liver pose estimation and replace FE-based deformation with non-rigid iterative closest point (NICP) to lower engineering/modeling complexity and expertise requirements. On real patient data, the depth-augmented foundation pose approach achieved 9.91 mm mean registration error in 3 cases. Combined rigid-NICP registration outperformed rigid-only registration, demonstrating NICP as an efficient substitute for finite-element deformable models. This pipeline achieves clinically relevant accuracy while offering a lightweight, engineering-friendly alternative to FE-based deformation.
Paper Structure (20 sections, 10 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 20 sections, 10 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORKS
  3. Rigid Initialization in Liver Laparoscopic Surgery
  4. Non-Rigid Registration in Liver Laparoscopic Surgery
  5. Pose Foundation
  6. METHOD
  7. Pose Offset Prediction
  8. Pre-Operative Non-rigid Data Collection
  9. Non-Rigid Optimization
  10. EXPERIMENTS
  11. Rigid Offest Prediction
  12. Data Generation
  13. Contour Augmentations.
  14. Mask Augmentation.
  15. Depth Augmentation
  16. ...and 5 more sections

Figures (3)

  • Figure 1: The registration workflow diagram: The first stage adopts the Refine Net component from PoseFoundation by Wen et al., with the inputs replaced by the liver’s contour, mask, and depth images. The output is the contour, mask, and depth images re-rendered using the predicted pose.
  • Figure 2: PCA-based liver deformation model. A reference liver mesh (source) is registered to multiple deformed liver configurations (targets) using NICP to establish vertex correspondence. montana2023saramis. PCA analysis of these aligned meshes extracts the first ten principal deformation modes. Linear combinations of these modes generate new plausible deformations, shown as red and green dashed contours, while the solid green contour represents the mean shape.
  • Figure 3: The registration results of the first-frame images for Patient 1, 3, and 4. The leftmost column shows the input intra-operative image. The second column presents the rigid pose estimation result obtained by iterative prediction using the FoundationPose model, overlaid on the intra-operative image. The rightmost column shows the result after further non-rigid refinement.