XRoboToolkit: A Cross-Platform Framework for Robot Teleoperation

TL;DR

This paper introduces XRoboToolkit, a cross-platform XR-based robot teleoperation framework built on OpenXR that enables low-latency stereoscopic feedback, optimization-based inverse kinematics, and dexterous hand retargeting across diverse robotic platforms. It integrates a Unity XR client with a Python/C++ backend, supporting multiple tracking modalities and simulators (e.g., MuJoCo, UR5, ARX R5, Galaxea R1-Lite) to facilitate real-time teleoperation and data collection for Vision-Language-Action models. The authors demonstrate versatile applications, including XR controller teleoperation, precision manipulation with active stereo vision, motion-tracker-guided redundant control, and MuJoCo hand control, and validate data quality by training VLA models that achieve autonomous performance. Limitations include whole-body tracking standardization, underactuated hand retargeting constraints, and MuJoCo-only simulation; future work targets hand retargeting improvements, multi-simulator support, humanoid teleoperation, and OpenXR standardization to enhance cross-platform compatibility.

Abstract

The rapid advancement of Vision-Language-Action models has created an urgent need for large-scale, high-quality robot demonstration datasets. Although teleoperation is the predominant method for data collection, current approaches suffer from limited scalability, complex setup procedures, and suboptimal data quality. This paper presents XRoboToolkit, a cross-platform framework for extended reality based robot teleoperation built on the OpenXR standard. The system features low-latency stereoscopic visual feedback, optimization-based inverse kinematics, and support for diverse tracking modalities including head, controller, hand, and auxiliary motion trackers. XRoboToolkit's modular architecture enables seamless integration across robotic platforms and simulation environments, spanning precision manipulators, mobile robots, and dexterous hands. We demonstrate the framework's effectiveness through precision manipulation tasks and validate data quality by training VLA models that exhibit robust autonomous performance.

Figures (6)

• Figure 1: Overview of XRoboToolkit, an integrative framework bridging XR and robotics. Core functionalities include real-time teleoperation and stereoscopic vision. Green blocks represent XR-side components, while blue blocks indicate components on the robot side, which runs on either the robot PC or a separate PC connected to the same network as the headset.
• Figure 2: OpenXR conventions for pose tracking coordinate system openxr_spec.
• Figure 3: Conventions for (a) hand tracking keypoints and (b) whole-body tracking keypoints.
• Figure 4: Screenshot of the PICO version of the XR Unity Application.
• Figure 5: Example applications of XRoboToolkit: (a) teleoperation with XR controllers for dual arm manipulation and mobile manipulators, (b) Dual UR5 manipulators with 2-DOF head tracking and stereo vision, (c) auxiliary motion trackers for robot elbow control in MeshCat visualization, and (d) dexterous hand tracking in MuJoCo simulation.