Multi-User Mobile Augmented Reality for Cardiovascular Surgical Planning

TL;DR

The paper introduces ARCollab, a first multi-user mobile AR system for collaborative cardiovascular surgical planning that replaces time-intensive 3D-printed heart models with shared, real-time AR interactions. Built on iOS using SwiftUI, RealityKit, and ARKit, ARCollab uses Bonjour and Multipeer Connectivity to form group sessions, supports multi-user model viewing with raycast anchoring, and enables novel omni-directional slicing through per-type transformation synchronization. Key innovations include double-sided polygon rendering to visualize hollow heart structures, plane-based slicing with the equation $ax+by+cz=d$ updated from finger gestures and synchronized across devices, and a save/load mechanism for model states. A usability evaluation with five medical experts demonstrated that ARCollab facilitates collaboration and is intuitive to use, highlighting portability, familiar gestures, and real-time shared views as strong benefits. The work demonstrates the practical potential of portable, collaborative mobile AR in surgical planning and provides an open-source platform for further research and extension.

Abstract

Collaborative planning for congenital heart diseases typically involves creating physical heart models through 3D printing, which are then examined by both surgeons and cardiologists. Recent developments in mobile augmented reality (AR) technologies have presented a viable alternative, known for their ease of use and portability. However, there is still a lack of research examining the utilization of multi-user mobile AR environments to support collaborative planning for cardiovascular surgeries. We created ARCollab, an iOS AR app designed for enabling multiple surgeons and cardiologists to interact with a patient's 3D heart model in a shared environment. ARCollab enables surgeons and cardiologists to import heart models, manipulate them through gestures and collaborate with other users, eliminating the need for fabricating physical heart models. Our evaluation of ARCollab's usability and usefulness in enhancing collaboration, conducted with three cardiothoracic surgeons and two cardiologists, marks the first human evaluation of a multi-user mobile AR tool for surgical planning. ARCollab is open-source, available at https://github.com/poloclub/arcollab.

Figures (4)

• Figure 1: Devices scan environment to detect feature points
• Figure 2: Gestures are used to manipulate the model. The orange arrows represent the gesture performed on one device, the white arrows represent resultant movement of the heart on other devices in the session, and the blue arrows represent actual manipulation of the model. (A) Pinch gesture enables the user to scale the heart model up or down. (B) Panning across the screen triggers rotation of the heart in the direction of the pan. (C) Two-finger rotation on the screen allows the user to rotate the heart around the axis pointing out of the camera perpendicular to the device.
• Figure 3: Our approach renders the "inner" surface of the heart in a darker color to give the appearance of "filling" hollow regions.
• Figure 4: Average ratings about ARCollab from the 2 cardiologists and 3 surgeons who participated in the study. All participants found ARCollab to facilitate collaborative surgical planning.