Ensuring Force Safety in Vision-Guided Robotic Manipulation via Implicit Tactile Calibration

TL;DR

The paper introduces SafeDiff, a diffusion-based framework that implicitly calibrates vision-guided robot states with real-time tactile feedback to ensure force safety in door-opening manipulation. It blends Vision-Guided Mapping Modules with Tactile-Guided Calibration Modules in a multi-scale diffusion network to produce safe state sequences conditioned on visual context and force signals. A large multimodal dataset, SafeDoorManip50k, supports training and evaluation, and a novel force-safety benchmark with simulation and real-world experiments demonstrates improved safety, robustness to disturbances, and effective sim-to-real transfer. The work provides a practical, data-driven approach to force-safe manipulation and offers new benchmarks and resources for future research.

Abstract

In dynamic environments, robots often encounter constrained movement trajectories when manipulating objects with specific properties, such as doors. Therefore, applying the appropriate force is crucial to prevent damage to both the robots and the objects. However, current vision-guided robot state generation methods often falter in this regard, as they lack the integration of tactile perception. To tackle this issue, this paper introduces a novel state diffusion framework termed SafeDiff. It generates a prospective state sequence from the current robot state and visual context observation while incorporating real-time tactile feedback to refine the sequence. As far as we know, this is the first study specifically focused on ensuring force safety in robotic manipulation. It significantly enhances the rationality of state planning, and the safe action trajectory is derived from inverse dynamics based on this refined planning. In practice, unlike previous approaches that concatenate visual and tactile data to generate future robot state sequences, our method employs tactile data as a calibration signal to adjust the robot's state within the state space implicitly. Additionally, we've developed a large-scale simulation dataset called SafeDoorManip50k, offering extensive multimodal data to train and evaluate the proposed method. Extensive experiments show that our visual-tactile model substantially mitigates the risk of harmful forces in the door opening, across both simulated and real-world settings.

Figures (5)

• Figure 1: The restoring force exerted by the robot’s end-effector can be decomposed into three components: $F_x$, $F_y$, and $F_z$. The component $F_z$ is tangent with the door’s opening trajectory and is termed the effective force. The forces lying in the xOy plane are orthogonal to the trajectory. These forces might cause damage to both the robot and the door and are referred to as harmful forces.
• Figure 2: Our framework takes a noise sequence as input, visual information, current robot state, and its corresponding force feedback as conditions and outputs the final safe states through $T$ denoising iterations. The architecture consists of an encoder and a decoder. The encoder is composed of a series of multi-scale Vision-Guided Mapping Modules (VMMs) that integrate visual data using FiLM perez2018film and generate state representations initially. The decoder comprises a stack of Tactile-Guided Calibration Modules (TCMs) which can refine the state representations based on tactile feedback.
• Figure 3: Qualitative results of our method in real-world scenarios. Each row corresponds to a specific door-opening task: The first row evaluates the effectiveness of our few-shot fine-tuning model in real-world settings (relevant to Q1), the second row assesses the model’s generalization capabilities (relevant to Q2), and the third row examines the model’s resistance to disturbances (relevant to Q3). Additionally, the first three columns in each row capture two samples from the door-opening process, while the final column quantifies the magnitude of harmful force encountered throughout the entire door-opening. Zoom in $10$ times for the better view.
• Figure 4: Sample of simulation environments
• Figure 5: Quantitative evaluation of different methods on our SafeDoorManip50k unseen-door scenarios with disturbance, highlighting the anti-disturbance capability of our method.