Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

5G Virtual Reality Manipulator Teleoperation using a Mobile Phone

Alexander Werner, William Melek

TL;DR

This work introduces a portable teleoperation pipeline that uses a consumer mobile phone as the leader device to drive a robot manipulator in $SE(3)$ by fusing the phone's IMU and camera data. It delivers immersive, multi-view visual feedback via VR-rendered point clouds and conveys interaction forces through the phone's haptic actuator, all over wireless networks such as 5G or Wi‑Fi. The system combines visual-inertial pose estimation, a compliant Cartesian impedance controller, and a momentum-based contact observer, with careful handling of bandwidth and latency through WebRTC and dedicated codecs. The results show the approach is viable for temporary or remote manipulation tasks, with 5G offering more consistent control than Wi‑Fi and visual feedback identified as a key bottleneck to further improvement.

Abstract

This paper presents an approach to teleoperate a manipulator using a mobile phone as a leader device. Using its IMU and camera, the phone estimates its Cartesian pose which is then used to to control the Cartesian pose of the robot's tool. The user receives visual feedback in the form of multi-view video - a point cloud rendered in a virtual reality environment. This enables the user to observe the scene from any position. To increase immersion, the robot's estimate of external forces is relayed using the phone's haptic actuator. Leader and follower are connected through wireless networks such as 5G or Wi-Fi. The paper describes the setup and analyzes its performance.

5G Virtual Reality Manipulator Teleoperation using a Mobile Phone

TL;DR

This work introduces a portable teleoperation pipeline that uses a consumer mobile phone as the leader device to drive a robot manipulator in by fusing the phone's IMU and camera data. It delivers immersive, multi-view visual feedback via VR-rendered point clouds and conveys interaction forces through the phone's haptic actuator, all over wireless networks such as 5G or Wi‑Fi. The system combines visual-inertial pose estimation, a compliant Cartesian impedance controller, and a momentum-based contact observer, with careful handling of bandwidth and latency through WebRTC and dedicated codecs. The results show the approach is viable for temporary or remote manipulation tasks, with 5G offering more consistent control than Wi‑Fi and visual feedback identified as a key bottleneck to further improvement.

Abstract

This paper presents an approach to teleoperate a manipulator using a mobile phone as a leader device. Using its IMU and camera, the phone estimates its Cartesian pose which is then used to to control the Cartesian pose of the robot's tool. The user receives visual feedback in the form of multi-view video - a point cloud rendered in a virtual reality environment. This enables the user to observe the scene from any position. To increase immersion, the robot's estimate of external forces is relayed using the phone's haptic actuator. Leader and follower are connected through wireless networks such as 5G or Wi-Fi. The paper describes the setup and analyzes its performance.
Paper Structure (11 sections, 10 equations, 5 figures)

This paper contains 11 sections, 10 equations, 5 figures.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. DESIGN
  4. CONTROL & FEEDBACK
  5. Visual feedback
  6. Command
  7. Manipulator Control
  8. Contact Feedback
  9. EXPERIMENTAL SETUP
  10. PERFORMANCE EVALUATION
  11. CONCLUSION

Figures (5)

  • Figure 1: Experimental setup used for evaluation.
  • Figure 2: Leader and follower system and transmitted data streams. Also shown: Relevant frames.
  • Figure 3: Evaluation setup: Manipulator and phone equipped with tracking markers.
  • Figure 4: Example user session captured together with the accompanying video. The top plot shows the position of the leader device in $z$ direction, the middle plot the follower position. The bottom plot shows the estimated external forces. In the upper two plots the shaded areas are periods when the clutch is activated, in the lower plot the shaded areas denote an active contact.
  • Figure 5: Tracking performance during a dynamic motion as recorded by the external tracking system. The user holds the clutch engaged during the area shaded in red.