Navigate Biopsy with Ultrasound under Augmented Reality Device: Towards Higher System Performance

TL;DR

The paper addresses the challenge of limited spatial perception in ultrasound-guided biopsies by introducing an AR navigation system that streams ultrasound images to a HoloLens 2 with minimal latency using remote rendering. It combines infrared tracking of both the ultrasound probe and biopsy needle, a registration approach that ties image space to AR space, and distinct in-plane/out-of-plane visualization cues to aid punctures. Key findings show end-to-end display latency around $122.49\pm11.61$ ms with an added AR latency of $16.22\pm11.45$ ms, and navigation accuracy of $1.23\pm0.68$ mm in-plane and $0.95\pm0.70$ mm out-of-plane, enabling substantial improvements in biopsy success rates in a use-case with novice operators (up to $98\%$ out-of-plane and $93\%$ in-plane). The approach demonstrates that real-time AR guidance can meaningfully speed up biopsies and improve accuracy, with practical implications for clinical adoption and future enhancements in 3D ultrasound visualization.

Abstract

Purpose: Biopsies play a crucial role in determining the classification and staging of tumors. Ultrasound is frequently used in this procedure to provide real-time anatomical information. Using augmented reality (AR), surgeons can visualize ultrasound data and spatial navigation information seamlessly integrated with real tissues. This innovation facilitates faster and more precise biopsy operations. Methods: We developed an AR biopsy navigation system with low display latency and high accuracy. Ultrasound data is initially read by an image capture card and streamed to Unity via net communication. In Unity, navigation information is rendered and transmitted to the HoloLens 2 device using holographic remoting. Retro-reflective tool tracking is implemented on the HoloLens 2, enabling simultaneous tracking of the ultrasound probe and biopsy needle. Distinct navigation information is provided during in-plane and out-of-plane punctuation. To evaluate the effectiveness of our system, we conducted a study involving ten participants, for puncture accuracy and biopsy time, comparing to traditional methods. Results: Our proposed framework enables ultrasound visualization in AR with only $16.22\pm11.45ms$ additional latency. Navigation accuracy reached $1.23\pm 0.68mm$ in the image plane and $0.95\pm 0.70mm$ outside the image plane. Remarkably, the utilization of our system led to $98\%$ and $95\%$ success rate in out-of-plane and in-plane biopsy. Conclusion: To sum up, this paper introduces an AR-based ultrasound biopsy navigation system characterized by high navigation accuracy and minimal latency. The system provides distinct visualization contents during in-plane and out-of-plane operations according to their different characteristics. Use case study in this paper proved that our system can help young surgeons perform biopsy faster and more accurately.

Figures (10)

• Figure 1: Biopsy navigation using ultrasound information overlayed on the real world. (a) Out-of-plane navigation. (b) In-plane navigation. Different visual contents are provided in these two modes to improve biopsy accuracy.
• Figure 2: Structure of proposed ultrasound biopsy navigation system. Ultrasound images are captured and processed by a high-performance computer. Net communication synchronizes ultrasound data to Unity, which is rendered and streamed to HoloLens 2 through holographic remoting. Infrared tool tracking is implemented on the AR headset.
• Figure 3: Registration of ultrasound image and infrared tool. The spaces of the ultrasound probe and infrared tool are aligned during designing. Transform from the ultrasound image to the probe is calculated using the shape of the valid imaging area.
• Figure 4: Visualization contents during in-plane punctuation. (a) Calculation of offset between biopsy needle and ultrasound image. Conversion from (b) rotation and (c) translation offset values to visualization status. Here, the color of the line indicates the display color of the target visual content.
• Figure 5: Visualize contents during out-of-plane punctuation. (a) Visual effect during out-of-plane navigation. (b) Conversion from the spatial relationship between the ultrasound probe and the biopsy needle to the status of visual contents.