Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual-Haptic Model Mediated Teleoperation for Remote Ultrasound

David Black, Maria Tirindelli, Septimiu Salcudean, Wolfgang Wein, Marco Esposito

TL;DR

This paper tackles the instability caused by time delays in tele-ultrasound by introducing visual-haptic model-mediated teleoperation (VH-MMT), which provides both instant local haptic feedback and a delay-free visual preview derived from a pre-recorded US sweep. The approach combines a RGB-D–based local scene model with a pre-acquired US volume re-sliced to align with the operator's haptic pose, enabling robust rough positioning despite network delays. A prototype using a 7-DoF robot, RGB-D sensing, and haptic feedback demonstrated that VH-MMT compensates delays up to 1000 ms RTT, significantly reducing operator effort and completion time and improving motion accuracy and pressure consistency compared with standard MMT. The findings suggest VH-MMT can make remote robotic ultrasound more practical, particularly for initial navigation and rough alignment in resource-constrained connectivity scenarios.

Abstract

Tele-ultrasound has the potential greatly to improve health equity for countless remote communities. However, practical scenarios involve potentially large time delays which cause current implementations of telerobotic ultrasound (US) to fail. Using a local model of the remote environment to provide haptics to the expert operator can decrease teleoperation instability, but the delayed visual feedback remains problematic. This paper introduces a robotic tele-US system in which the local model is not only haptic, but also visual, by re-slicing and rendering a pre-acquired US sweep in real time to provide the operator a preview of what the delayed image will resemble. A prototype system is presented and tested with 15 volunteer operators. It is found that visual-haptic model-mediated teleoperation (MMT) compensates completely for time delays up to 1000 ms round trip in terms of operator effort and completion time while conventional MMT does not. Visual-haptic MMT also significantly outperforms MMT for longer time delays in terms of motion accuracy and force control. This proof-of-concept study suggests that visual-haptic MMT may facilitate remote robotic tele-US.

Visual-Haptic Model Mediated Teleoperation for Remote Ultrasound

TL;DR

This paper tackles the instability caused by time delays in tele-ultrasound by introducing visual-haptic model-mediated teleoperation (VH-MMT), which provides both instant local haptic feedback and a delay-free visual preview derived from a pre-recorded US sweep. The approach combines a RGB-D–based local scene model with a pre-acquired US volume re-sliced to align with the operator's haptic pose, enabling robust rough positioning despite network delays. A prototype using a 7-DoF robot, RGB-D sensing, and haptic feedback demonstrated that VH-MMT compensates delays up to 1000 ms RTT, significantly reducing operator effort and completion time and improving motion accuracy and pressure consistency compared with standard MMT. The findings suggest VH-MMT can make remote robotic ultrasound more practical, particularly for initial navigation and rough alignment in resource-constrained connectivity scenarios.

Abstract

Tele-ultrasound has the potential greatly to improve health equity for countless remote communities. However, practical scenarios involve potentially large time delays which cause current implementations of telerobotic ultrasound (US) to fail. Using a local model of the remote environment to provide haptics to the expert operator can decrease teleoperation instability, but the delayed visual feedback remains problematic. This paper introduces a robotic tele-US system in which the local model is not only haptic, but also visual, by re-slicing and rendering a pre-acquired US sweep in real time to provide the operator a preview of what the delayed image will resemble. A prototype system is presented and tested with 15 volunteer operators. It is found that visual-haptic model-mediated teleoperation (MMT) compensates completely for time delays up to 1000 ms round trip in terms of operator effort and completion time while conventional MMT does not. Visual-haptic MMT also significantly outperforms MMT for longer time delays in terms of motion accuracy and force control. This proof-of-concept study suggests that visual-haptic MMT may facilitate remote robotic tele-US.

Paper Structure

This paper contains 11 sections, 2 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. System Overview
  4. Prototype System
  5. Robot Control
  6. Communication
  7. Calibration
  8. Haptics
  9. Experiments
  10. Results
  11. Discussion and Conclusion

Figures (4)

  • Figure 1: The local visual and haptic models shown to the expert during teleoperation. (A) shows the hand-eye and phantom calibration which allows the local visual and haptic models to be correctly co-located (C). The predicted US plane (red square in (C)) is used to re-slice the US sweep at any angle, generating the preview image in (B). Two orthogonal slices and a 3D view of the example sweep are shown in (D).
  • Figure 2: Overview of the visual-haptic MMT system. The "live" streams here are those sent directly to the expert console, so they may be significantly delayed.
  • Figure 3: Point cloud-based haptics for model-mediated teleoperation. The cube represents an octree, which is used to search the point cloud. A surface is fitted to a neighborhood of points (green circle), and a virtual spring-damper is used to compute the force.
  • Figure 4: Plan of US scanning task on branching vessel phantom. Steps 1-5 are outlined in the text, and each view is shown in the US below. The red lines are guides for the user to keep the vessels centered. The ArUco marker edge indicated by the red arrow shows where the user must switch between the longitudinal and transverse views.