Adaptive Anomaly Recovery for Telemanipulation: A Diffusion Model Approach to Vision-Based Tracking

TL;DR

This work tackles the challenge of unstable vision-based telemanipulation under visual anomalies by introducing Diffusion-Enhanced Telemanipulation (DET). DET combines Frame-Difference Detection (FDD) to segment anomalous video segments with diffusion-based reconstruction to restore high-dimensional frames, which are then used as targets for a DRL controller within a Markov Game framework for telemanipulation. The approach is evaluated against low-dimensional baselines (Fourier and cubic spline) across multiple anomaly types, speeds, and durations, using a diffusion model (Sora) for reconstruction and ArUco-based pose estimation. Results show DET improves robustness and maintains near-ground-truth robot performance under challenging visual conditions, with indications of feasibility for real-time deployment on local hardware.

Abstract

Dexterous telemanipulation critically relies on the continuous and stable tracking of the human operator's commands to ensure robust operation. Vison-based tracking methods are widely used but have low stability due to anomalies such as occlusions, inadequate lighting, and loss of sight. Traditional filtering, regression, and interpolation methods are commonly used to compensate for explicit information such as angles and positions. These approaches are restricted to low-dimensional data and often result in information loss compared to the original high-dimensional image and video data. Recent advances in diffusion-based approaches, which can operate on high-dimensional data, have achieved remarkable success in video reconstruction and generation. However, these methods have not been fully explored in continuous control tasks in robotics. This work introduces the Diffusion-Enhanced Telemanipulation (DET) framework, which incorporates the Frame-Difference Detection (FDD) technique to identify and segment anomalies in video streams. These anomalous clips are replaced after reconstruction using diffusion models, ensuring robust telemanipulation performance under challenging visual conditions. We validated this approach in various anomaly scenarios and compared it with the baseline methods. Experiments show that DET achieves an average RMSE reduction of 17.2% compared to the cubic spline and 51.1% compared to FFT-based interpolation for different occlusion durations.

Figures (3)

• Figure 1: We model the telemanipulation tasks as a Markov Game that involves two agents: a human operator and a robot. The robot’s MDP depends on the operation command from the state of the human’s POMDP because the human’s mind is a black box and only the end effects that the human applied to the object can be observed by the robot.
• Figure 2: The offline data collection setup involves a human operator rotating a block with an ArUco marker, which is recorded by an overhead camera. The IMU provides ground truth rotation data. Added anomalies will disrupt the visibility of the marker. Then, the anomalous segments are reconstructed using the DET framework. These reconstructed frames are then used to control the robot in real time, and the resulting performance is compared against the ground truth data for evaluation.
• Figure 3: Comparison of DET and baseline methods under different Lighting Change durations. DET achieved better performance in all conditions