Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tactile-Conditioned Diffusion Policy for Force-Aware Robotic Manipulation

Erik Helmut, Niklas Funk, Tim Schneider, Cristiana de Farias, Jan Peters

TL;DR

This work introduces FARM, a tactile-conditioned diffusion policy for force-aware robotic manipulation. By fusing high-dimensional tactile information from GelSight sensors with explicit grip-force commands, FARM learns to predict and control both grip width and grip force in a unified diffusion framework, enabling stable contact-rich manipulation across tasks with diverse force requirements. Real-robot experiments demonstrate that explicit force signals and high-dimensional force distributions substantially improve success rates, especially for delicate or dynamic contact tasks, and that force-distribution knowledge yields closer alignment to demonstration forces as shown by Wasserstein analyses. The approach closes the loop between tactile sensing and force control, offering a practical, transferable pipeline from demo collection with a hand-held gripper to deployment on an actuated robotic hand, with open-source hardware and software accompanying the work.

Abstract

Contact-rich manipulation depends on applying the correct grasp forces throughout the manipulation task, especially when handling fragile or deformable objects. Most existing imitation learning approaches often treat visuotactile feedback only as an additional observation, leaving applied forces as an uncontrolled consequence of gripper commands. In this work, we present Force-Aware Robotic Manipulation (FARM), an imitation learning framework that integrates high-dimensional tactile data to infer tactile-conditioned force signals, which in turn define a matching force-based action space. We collect human demonstrations using a modified version of the handheld Universal Manipulation Interface (UMI) gripper that integrates a GelSight Mini visual tactile sensor. For deploying the learned policies, we developed an actuated variant of the UMI gripper with geometry matching our handheld version. During policy rollouts, the proposed FARM diffusion policy jointly predicts robot pose, grip width, and grip force. FARM outperforms several baselines across three tasks with distinct force requirements -- high-force, low-force, and dynamic force adaptation -- demonstrating the advantages of its two key components: leveraging force-grounded, high-dimensional tactile observations and a force-based control space. The codebase and design files are open-sourced and available at https://tactile-farm.github.io .

Tactile-Conditioned Diffusion Policy for Force-Aware Robotic Manipulation

TL;DR

This work introduces FARM, a tactile-conditioned diffusion policy for force-aware robotic manipulation. By fusing high-dimensional tactile information from GelSight sensors with explicit grip-force commands, FARM learns to predict and control both grip width and grip force in a unified diffusion framework, enabling stable contact-rich manipulation across tasks with diverse force requirements. Real-robot experiments demonstrate that explicit force signals and high-dimensional force distributions substantially improve success rates, especially for delicate or dynamic contact tasks, and that force-distribution knowledge yields closer alignment to demonstration forces as shown by Wasserstein analyses. The approach closes the loop between tactile sensing and force control, offering a practical, transferable pipeline from demo collection with a hand-held gripper to deployment on an actuated robotic hand, with open-source hardware and software accompanying the work.

Abstract

Contact-rich manipulation depends on applying the correct grasp forces throughout the manipulation task, especially when handling fragile or deformable objects. Most existing imitation learning approaches often treat visuotactile feedback only as an additional observation, leaving applied forces as an uncontrolled consequence of gripper commands. In this work, we present Force-Aware Robotic Manipulation (FARM), an imitation learning framework that integrates high-dimensional tactile data to infer tactile-conditioned force signals, which in turn define a matching force-based action space. We collect human demonstrations using a modified version of the handheld Universal Manipulation Interface (UMI) gripper that integrates a GelSight Mini visual tactile sensor. For deploying the learned policies, we developed an actuated variant of the UMI gripper with geometry matching our handheld version. During policy rollouts, the proposed FARM diffusion policy jointly predicts robot pose, grip width, and grip force. FARM outperforms several baselines across three tasks with distinct force requirements -- high-force, low-force, and dynamic force adaptation -- demonstrating the advantages of its two key components: leveraging force-grounded, high-dimensional tactile observations and a force-based control space. The codebase and design files are open-sourced and available at https://tactile-farm.github.io .
Paper Structure (20 sections, 1 equation, 6 figures)

This paper contains 20 sections, 1 equation, 6 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Incorporating Visuotactile Observations into Robotic Manipulation
  4. Tactile Sensing for Action Representation
  5. Method
  6. Gripper Hardware
  7. Data Collection
  8. FARM Diffusion Policy
  9. Policy Deployment and Gripper Control
  10. Baselines
  11. Force-Aware Baseline
  12. Tactile-Aware Baseline
  13. Vision-Only Baseline
  14. Experiments & Results
  15. Experiment Setup
  16. ...and 5 more sections

Figures (6)

  • Figure 1: Data-collection setup using the adapted hand-held UMI gripper. Right: An expert performs a task using the adapted hand-held UMI gripper. Top left: In-hand RGB camera view with ArUco markers for grip width measurement. Bottom left: GelSight Mini tactile image and corresponding FEATS force estimates, visualizing the contact interaction and force distributions during demonstration.
  • Figure 2: Side-by-side comparison of the adapted umi gripper (left) used for demonstration data collection, and the Actuated UMI gripper (right) used for robotic deployment. Both designs feature an Intel RealSense D405 camera, a GelSight Mini tactile sensor (mounted on one fingertip), OptiTrack markers for motion tracking, and ArUco markers for grip width measurement. Sensor placement and overall geometry are matched to enable direct transfer of learned policies to the Actuated UMI.
  • Figure 3: Schematic of the farm diffusion policy architecture. Visual, proprioceptive, and tactile observations are encoded and provided as input to a 1D temporal CNN with FiLM conditioning. The model predicts action trajectories including absolute end-effector pose, grip width, and grip force. This structure enables closed-loop force control of the gripper during manipulation.
  • Figure 4: Experiment setup for the three tasks: (a) plant insertion, showing the Franka Research 3 robot planting a plastic plant into a soil-filled pot; (b) grape picking, depicting the delicate grasping and detachment of a grape from a toothpick; (c) screw tightening, illustrating the robot using an Allen key to tighten a screw.
  • Figure 5: Success rates for the plant insertion, grape picking, and screw tightening tasks, comparing the farm framework against force-aware, tactile-aware, and vision-only baselines, evaluated over 20 rollouts per task.
  • ...and 1 more figures