ARCollab: Towards Multi-User Interactive Cardiovascular Surgical Planning in Mobile Augmented Reality

TL;DR

ARCollab targets collaborative cardiovascular surgical planning in mobile AR, addressing the lack of multi-user, patient-specific 3D heart visualization without physical models. The approach integrates ARKit/RealityKit on iOS with MultipeerConnectivity and cloud-based imports (OneDrive) to support shared sessions, real-time gesture-driven manipulation, and cross-device synchronization via transformation representations. The main contributions are (i) a first-of-its-kind multi-user mobile AR design for surgical planning, (ii) a gesture-to-transformation pipeline that encodes rotations as quaternions $q$ applied to an orientation matrix $M$, with per-gesture updates to peers, and (iii) support for importing patient-specific data from HIPAA-compliant clouds and sharing it across peers. Ongoing work includes collaborative model slicing, usability evaluation with surgeons/cardiologists, and deployment via the iOS App Store to enable broad adoption.

Abstract

Surgical planning for congenital heart diseases requires a collaborative approach, traditionally involving the 3D-printing of physical heart models for inspection by surgeons and cardiologists. Recent advancements in mobile augmented reality (AR) technologies have offered a promising alternative, noted for their ease-of-use and portability. Despite this progress, there remains a gap in research exploring the use of multi-user mobile AR environments for facilitating collaborative cardiovascular surgical planning. We are developing ARCollab, an iOS AR application designed to allow multiple surgeons and cardiologists to interact with patient-specific 3D heart models in a shared environment. ARCollab allows surgeons and cardiologists to import heart models, perform gestures to manipulate the heart, and collaborate with other users without having to produce a physical heart model. We are excited by the potential for ARCollab to make long-term real-world impact, thanks to the ubiquity of iOS devices that will allow for ARCollab's easy distribution, deployment and adoption.

Figures (4)

• Figure 1: An example of a 3D-printed heart model
• Figure 2: (A) To initiate a shared session, two surgeons can use their devices to scan their physical environment to detect feature points in their surroundings (shown as green dots), which are notable environmental elements that a device can consistently track across frames, such as the corner of a wooden table. (B) Once the feature points are shared over the network, a pop-up notification is used to inform the user that the AR environments are aligned and the collaborative experience is established.
• Figure 3: Gestures are used to manipulate the model. The orange arrows represent the gesture performed on one device, the white arrows represent resultant movement of the heart on other devices in the session, and the blue arrows represent actual manipulation of the model. (a) The user can perform a pinch gesture to scale the heart model up or down. (b) The user can pan across the screen with their finger to trigger a rotation of the heart in the direction of the pan.
• Figure 4: The surgeons and cardiologists can select a 3D model from their HIPAA-compliant OneDrive storage options and the model will be shared to other devices on the network so that they can render it.