Shared Telemanipulation with VR controllers in an anti slosh scenario

TL;DR

This work introduces a VR-based telemanipulation framework driven by nonlinear model predictive control (NMPC) to address delays, limited feedback, and high haptic costs in remote manipulation. By modeling both the robot arm (UR5e) and the liquid slosh as a pendulum-like dynamic, the NMPC jointly optimizes trajectory tracking and anti-slosh stabilization, using a VR controller as the human input interface. The approach is implemented in ROS2/CasADi and validated in real time (≈$21$ ms per solve) on a UR5e with a glass of water, demonstrating a clear trade-off between high-fidelity tracking and anti-slosh performance and outlining extensions to pouring and collision avoidance. The results indicate that VR-based NMPC can provide intuitive, stable teleoperation for liquid handling tasks in elder care and related domains, with practical implications for low-cost, high-immersion remote manipulation.

Abstract

Telemanipulation has become a promising technology that combines human intelligence with robotic capabilities to perform tasks remotely. However, it faces several challenges such as insufficient transparency, low immersion, and limited feedback to the human operator. Moreover, the high cost of haptic interfaces is a major limitation for the application of telemanipulation in various fields, including elder care, where our research is focused. To address these challenges, this paper proposes the usage of nonlinear model predictive control for telemanipulation using low-cost virtual reality controllers, including multiple control goals in the objective function. The framework utilizes models for human input prediction and taskrelated models of the robot and the environment. The proposed framework is validated on an UR5e robot arm in the scenario of handling liquid without spilling. Further extensions of the framework such as pouring assistance and collision avoidance can easily be included.

Figures (6)

• Figure 1: Planar robot arm with three rotational joints $q_i$ and a glass of water with the coordinate frame $\{4\}$ connected to the end effector.
• Figure 2: The liquid in a container is modelled as a pendulum.
• Figure 3: Mapping of user input to desired end effector position. A movement to the right an downwards of the VR controller moves the desired position of the end effector in positive $\mathbf{e}_x$ and negative $\mathbf{e}_z$ direction.
• Figure 4: Overview of the control architecture.
• Figure 5: Plots of the two parametrizations $\mathrm{P}_1$ and $\mathrm{P}_2$. Depicted are the reference and the actual trajectories in the task space as well as the calculated slosh angle $\beta$.