Robotic Arm Platform for Multi-View Image Acquisition and 3D Reconstruction in Minimally Invasive Surgery

TL;DR

The robotic platform provides a tool for controlled, repeatable multi-view data acquisition for 3D generation in MIS environments, which can lead to new datasets necessary for novel learning-based surgical models.

Abstract

Minimally invasive surgery (MIS) offers significant benefits such as reduced recovery time and minimised patient trauma, but poses challenges in visibility and access, making accurate 3D reconstruction a significant tool in surgical planning and navigation. This work introduces a robotic arm platform for efficient multi-view image acquisition and precise 3D reconstruction in MIS settings. We adapted a laparoscope to a robotic arm and captured ex-vivo images of several ovine organs across varying lighting conditions (operating room and laparoscopic) and trajectories (spherical and laparoscopic). We employed recently released learning-based feature matchers combined with COLMAP to produce our reconstructions. The reconstructions were evaluated against high-precision laser scans for quantitative evaluation. Our results show that whilst reconstructions suffer most under realistic MIS lighting and trajectory, many versions of our pipeline achieve close to sub-millimetre accuracy with an average of 1.05 mm Root Mean Squared Error and 0.82 mm Chamfer distance. Our best reconstruction results occur with operating room lighting and spherical trajectories. Our robotic platform provides a tool for controlled, repeatable multi-view data acquisition for 3D generation in MIS environments which we hope leads to new datasets for training learning-based models.

Figures (5)

• Figure 1: This work creates reconstructions from multi-view images collected by our robotic arm platform. We use a laser scanner to obtain ground truth data for comparison.
• Figure 2: a) Annotated figure showing the robotic platform. The robot is attached to the table using a custom mount. The laparoscope optical system is attached to the robot with printed and laser-cut parts. All trajectory control and data capture are handled by an Intel NUC. b) Depiction of acquisition setup and the three trajectories: Trocar, Open-Close and Open-Far. Additionally, the sampling points in are depicted for both equal angle poses where poses are calculated by sequentially incrementing the azimuth and altitude angles by fixed amounts and Fibonacci sphere poses which are more even coverage over the imaging sphere.
• Figure 3: a) An frame from each of the 6 organ sample sets. Row 1&2 show different kidney samples. In row 3 we show a section of two liver samples. b) Data acquisition: A summary of all the data collected by our platform. Reconstruction: Our reconstruction pipeline for processing our data. Results: Evaluation of the reconstructions by comparing to acquired ground truth data.
• Figure 4: 3D reconstructions of a kidney and liver obtained using different trajectories and lighting conditions. For the kidney (row 1), data was captured using the Open-Far trajectory under operating room lighting, and processed with the ALIKED- image matcher. The liver (row 2) was captured using the trocar trajectory under laparoscopic lighting and processed using the GIM- matcher. In both cases, the predicted models (col 1&2) are compared to the ground truth laser scans (col 3) and the post-processed, aligned reconstructions (col 4).
• Figure 5: A graph depicting a sub-sampled set of poses of the ground truth (blue) and predicted (yellow) poses for a kidney dataset with the OR light source and the open-far trajectory using the ALIKED-LG image matcher.