Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward a Predictive eXtended Reality Teleoperation System with Duo-Virtual Spaces

Ziliang Zhang, Cong Liu, Hyoseung Kim

TL;DR

This paper tackles the high end-to-end latency in XR teleoperation systems, which impairs rapid maneuvers and precise manipulation in remote robot control. It proposes a duo-virtual-spaces architecture that localizes the remote agent and objects within the user-side virtual space and periodically calibrates with ground-truth poses from the agent, thereby substantially reducing latency. A latency profiling study and a case study show end-to-end latency reductions up to $89\%$ and identify pose-drift challenges when merging predicted and ground-truth states. The work demonstrates a practical pathway to predictive XR teleoperation with reduced dependence on network round-trip times, enabling safer and more responsive control in hazardous or inaccessible environments.

Abstract

Extended Reality (XR) provides a more intuitive interaction method for teleoperating robots compared to traditional 2D controls. Recent studies have laid the groundwork for usable teleoperation with XR, but it fails in tasks requiring rapid motion and precise manipulations due to the large delay between user motion and agent feedback. In this work, we profile the end-to-end latency in a state-of-the-art XR teleoperation system and propose our idea to optimize the latency by implementing a duo-virtual spaces design and localizing the agent and objects in the user-side virtual space, while calibrating with periodic ground-truth poses from the agent-side virtual space.

Toward a Predictive eXtended Reality Teleoperation System with Duo-Virtual Spaces

TL;DR

This paper tackles the high end-to-end latency in XR teleoperation systems, which impairs rapid maneuvers and precise manipulation in remote robot control. It proposes a duo-virtual-spaces architecture that localizes the remote agent and objects within the user-side virtual space and periodically calibrates with ground-truth poses from the agent, thereby substantially reducing latency. A latency profiling study and a case study show end-to-end latency reductions up to and identify pose-drift challenges when merging predicted and ground-truth states. The work demonstrates a practical pathway to predictive XR teleoperation with reduced dependence on network round-trip times, enabling safer and more responsive control in hazardous or inaccessible environments.

Abstract

Extended Reality (XR) provides a more intuitive interaction method for teleoperating robots compared to traditional 2D controls. Recent studies have laid the groundwork for usable teleoperation with XR, but it fails in tasks requiring rapid motion and precise manipulations due to the large delay between user motion and agent feedback. In this work, we profile the end-to-end latency in a state-of-the-art XR teleoperation system and propose our idea to optimize the latency by implementing a duo-virtual spaces design and localizing the agent and objects in the user-side virtual space, while calibrating with periodic ground-truth poses from the agent-side virtual space.
Paper Structure (5 sections, 3 figures)

This paper contains 5 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Characterizing Teleoperation Workload
  4. Challenges of latency optimization
  5. Conclusion

Figures (3)

  • Figure 1: Comparison between existing teleoperation and our proposed system. Our design conducts local prediction in XR for remote agent and objects, mitigating large network delay.
  • Figure 2: ROS Reality whitney2018ros, an open-sourced XR teleoperation system, implements a publisher-subscriber execution model for XR and uses ROS for agent. Agent depends on user poses to transform and XR depends on video, point cloud, and transform string to update the virtual space.
  • Figure 3: Latency profiling and challenges during poses merge.