An objective comparison of methods for augmented reality in laparoscopic liver resection by preoperative-to-intraoperative image fusion

TL;DR

The Preoperative-to-Intraoperative Laparoscopic Fusion challenge (P2ILF) is presented, which investigates the possibilities of detecting anatomical landmarks automatically and using them in registration and proposes three key hypotheses that determine current limitations and future directions for research in this domain.

Abstract

Augmented reality for laparoscopic liver resection is a visualisation mode that allows a surgeon to localise tumours and vessels embedded within the liver by projecting them on top of a laparoscopic image. Preoperative 3D models extracted from CT or MRI data are registered to the intraoperative laparoscopic images during this process. In terms of 3D-2D fusion, most of the algorithms make use of anatomical landmarks to guide registration. These landmarks include the liver's inferior ridge, the falciform ligament, and the occluding contours. They are usually marked by hand in both the laparoscopic image and the 3D model, which is time-consuming and may contain errors if done by a non-experienced user. Therefore, there is a need to automate this process so that augmented reality can be used effectively in the operating room. We present the Preoperative-to-Intraoperative Laparoscopic Fusion Challenge (P2ILF), held during the Medical Imaging and Computer Assisted Interventions (MICCAI 2022) conference, which investigates the possibilities of detecting these landmarks automatically and using them in registration. The challenge was divided into two tasks: 1) A 2D and 3D landmark detection task and 2) a 3D-2D registration task. The teams were provided with training data consisting of 167 laparoscopic images and 9 preoperative 3D models from 9 patients, with the corresponding 2D and 3D landmark annotations. A total of 6 teams from 4 countries participated, whose proposed methods were evaluated on 16 images and two preoperative 3D models from two patients. All the teams proposed deep learning-based methods for the 2D and 3D landmark segmentation tasks and differentiable rendering-based methods for the registration task. Based on the experimental outcomes, we propose three key hypotheses that determine current limitations and future directions for research in this domain.

Figures (8)

• Figure 1: Laparoscopic image fusion with preoperative 3D CT or MRI scans. A preoperative 3D scan is first used to reconstruct the liver boundaries, tumours and major vessels critical for a safe surgery. During the laparoscopic procedure we overlay the reconstructed model using image registration, in this case 3D meshes, to the 2D liver view. The idea is to project 3D mesh points onto the liver boundaries that can enable understanding of the spatial location of the tumours and vessels along with the matched liver boundaries in the acquired 3D model. Such an augmented reality technique helps surgeons to locate the tumour and important landmarks during surgery. The above results were obtained with the semi-automatic method from koo2017.
• Figure 2: Depiction of the 2D and 3D anatomical landmarks. Anatomical liver landmark ground-truth annotations in the preoperative 3D model (left), and in the laparoscopic 2D image (right).
• Figure 3: P2ILF dataset: Training and test data samples with original laparoscopic images, annotated anatomical landmarks (silhouette in yellow, ridge in red and falciform ligament in blue), and the corresponding 3D anatomical annotations (rigde in red and falciform ligament in blue) in manually aligned 3D liver models are provided. The dataset contains a total of 11 patients, divided in 9 patients for training and 2 patients for testing.
• Figure 4: Submission procedure of the P2ILF Teamchallenge: A docker container system for submission was established on the Grand Challenge platform. Each liver model and corresponding images together with intrinsic camera parameters were provided to the challenge participants. The algorithmic submission required different inputs for the prediction of 2D liver landmarks, 3D liver landmarks, and the use of these landmarks for registration of the 3D model to the laparoscopic images. Finally, the outputs from each team's algorithm were evaluated using different metrics (see the section Evaluation Metrics for more details).
• Figure 5: General pipeline of the six team methods.Team BHL: The input 2D image and 3D model are first processed and augmented. Two ResUNets are used to segment the 2D landmarks in the images, and one PointNet++ is used to segment the 3D landmarks in the preoperative 3D model. To perform 3D-2D registration, the correspondences are fed to the P$n$P algorithm and a transformation matrix is obtained. Team GRASP: Mask-RCNN is used to generate a 2D mask of the liver, which is then used to perform 3D-2D registration by minimising a silhouette reprojection error through differentiable rendering. Team NCT: nnUNet and MeshCNN are used to segment the 2D and 3D landmarks, respectively. Differential rendering is then used to perform 3D-2D registration by minimising a reprojection error of the previously detected landmarks. Team UCL: UNet++ is used to segment the 2D landmarks, while PointNet++ is used to segment the 3D landmarks. This team also used differential rendering to perform 3D-2D registration. Team VOR: The 2D case is treated as a pixel segmentation task and the 3D case as a vertex classification task. Differentiable rendering is then used to perform 3D-2D registration by generating 2D images from the affine transformations computed by the localisation networks. The shape regularisation terms provide extra supervision to avoid undesired mesh deformations. Team VIP: The team only participated in task 1. Attention UNet was used for the pixel segmentation task of the anatomical liver landmarks in the laparoscopic images.