Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TacUMI: A Multi-Modal Universal Manipulation Interface for Contact-Rich Tasks

Tailai Cheng, Kejia Chen, Lingyun Chen, Liding Zhang, Yue Zhang, Yao Ling, Mahdi Hamad, Zhenshan Bing, Fan Wu, Karan Sharma, Alois Knoll

TL;DR

TacUMI tackles the challenge of learning long-horizon, contact-rich manipulation by marrying a multi-modal handheld data collector with a BiLSTM-based segmentation framework. The hardware integrates tactile sensing (ViTac), a 6-axis force-torque sensor, and drift-free 6-DoF pose tracking in a robot-compatible form factor, together with a continuous locking mechanism to ensure clean interaction data. The segmentation model fuses tactile, visual, F/T, and pose streams and uses sliding windows and soft voting to robustly partition demonstrations into semantically meaningful skills, achieving over 90% accuracy on cable mounting. The results show strong transfer of models trained on TacUMI data to robot-operated demonstrations, highlighting TacUMI's potential for scalable multimodal imitation learning in complex manipulation tasks.

Abstract

Task decomposition is critical for understanding and learning complex long-horizon manipulation tasks. Especially for tasks involving rich physical interactions, relying solely on visual observations and robot proprioceptive information often fails to reveal the underlying event transitions. This raises the requirement for efficient collection of high-quality multi-modal data as well as robust segmentation method to decompose demonstrations into meaningful modules. Building on the idea of the handheld demonstration device Universal Manipulation Interface (UMI), we introduce TacUMI, a multi-modal data collection system that integrates additionally ViTac sensors, force-torque sensor, and pose tracker into a compact, robot-compatible gripper design, which enables synchronized acquisition of all these modalities during human demonstrations. We then propose a multi-modal segmentation framework that leverages temporal models to detect semantically meaningful event boundaries in sequential manipulations. Evaluation on a challenging cable mounting task shows more than 90 percent segmentation accuracy and highlights a remarkable improvement with more modalities, which validates that TacUMI establishes a practical foundation for both scalable collection and segmentation of multi-modal demonstrations in contact-rich tasks.

TacUMI: A Multi-Modal Universal Manipulation Interface for Contact-Rich Tasks

TL;DR

TacUMI tackles the challenge of learning long-horizon, contact-rich manipulation by marrying a multi-modal handheld data collector with a BiLSTM-based segmentation framework. The hardware integrates tactile sensing (ViTac), a 6-axis force-torque sensor, and drift-free 6-DoF pose tracking in a robot-compatible form factor, together with a continuous locking mechanism to ensure clean interaction data. The segmentation model fuses tactile, visual, F/T, and pose streams and uses sliding windows and soft voting to robustly partition demonstrations into semantically meaningful skills, achieving over 90% accuracy on cable mounting. The results show strong transfer of models trained on TacUMI data to robot-operated demonstrations, highlighting TacUMI's potential for scalable multimodal imitation learning in complex manipulation tasks.

Abstract

Task decomposition is critical for understanding and learning complex long-horizon manipulation tasks. Especially for tasks involving rich physical interactions, relying solely on visual observations and robot proprioceptive information often fails to reveal the underlying event transitions. This raises the requirement for efficient collection of high-quality multi-modal data as well as robust segmentation method to decompose demonstrations into meaningful modules. Building on the idea of the handheld demonstration device Universal Manipulation Interface (UMI), we introduce TacUMI, a multi-modal data collection system that integrates additionally ViTac sensors, force-torque sensor, and pose tracker into a compact, robot-compatible gripper design, which enables synchronized acquisition of all these modalities during human demonstrations. We then propose a multi-modal segmentation framework that leverages temporal models to detect semantically meaningful event boundaries in sequential manipulations. Evaluation on a challenging cable mounting task shows more than 90 percent segmentation accuracy and highlights a remarkable improvement with more modalities, which validates that TacUMI establishes a practical foundation for both scalable collection and segmentation of multi-modal demonstrations in contact-rich tasks.
Paper Structure (25 sections, 4 equations, 8 figures, 2 tables)

This paper contains 25 sections, 4 equations, 8 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Data Collection Interfaces for Robot
  4. Event Segmentation
  5. METHODOLOGY
  6. Mechanical Design for Robot-Executable Demonstrations
  7. Robot-Compatible Force–Torque Stack
  8. Rack-and-Pinion Trigger with Continuous Self-Locking
  9. ViTac Sensing for Contact-Rich Transitions
  10. Drift-Free 6-DoF Pose Tracking
  11. Event Segmentation
  12. Network Architecture
  13. Sliding Window and Training Strategy
  14. Soft Voting for Inference
  15. F/T Data Preprocessing
  16. ...and 10 more sections

Figures (8)

  • Figure 1: Overall setup for the cable mounting task. The left hand holds a modified UMI gripper equipped with a ViTac sensor and a Vive Tracker, while the right hand holds the proposed TacUMI gripper for multimodal data collection. A camera positioned in the center records third-person visual information during the demonstrations.
  • Figure 2: TacUMI structure. Exploded view of the proposed multimodal data collection gripper, showing: (1) ViTac sensor (GelSight Mini); (2) Vive tracker for 6-DoF pose tracking; (3) Trigger connected to the rack-and-pinion mechanism for grasp actuation; (4) Continuous locking part that enables the trigger to be locked at arbitrary jaw opening widths; (5) 6-axis Force/Torque (F/T) sensor; (6) Handle ergonomically designed for single-handed operation. Insets illustrate the locking mechanism in the locked and released states.
  • Figure 3: Pipeline of the proposed event segmentation algorithm. The framework consists of four main stages: (a) Data Extraction: tactile images are processed by a ResNet50 to obtain 256-dimensional embeddings; third-person view images are processed by a ResNet18 with GroupNorm to obtain another 256-dimensional visual embedding; raw 6D F/T signals are preprocessed and 6D tracker pose are transformed to TCP. (b) Sliding Window: fused feature sequences of size $[T,532]$ are segmented into overlapping windows of length 50 with stride 10. (c) Training: each window is fed into one of three sequence models (BiLSTM, TCN, or Transformer) to capture temporal dependencies and produce per-frame skill predictions. (d) soft voting inference: overlapping predictions for the same frame are aggregated by averaging class probabilities and assigning the highest-probability label, restoring the original sequence length for final output.
  • Figure 4: Comparison of original and processed Force/Torque (F/T) data. (a) Original F/T data: The recorded signals reveal distinct operator-induced events, namely pulling and locking the trigger (blue-shaded region) followed by releasing it (green-shaded region), both of which cause significant disturbances in the measured force and torque signals. (b) Processed F/T data: The influence of these three actions has been removed, yielding clean signals that can be directly utilized by the robot without the undesired human-induced artifacts.
  • Figure 5: Overview of the cable mounting task process. Using the proposed TacUMI, the cable is sequentially inserted into three U-type clips with different orientations.
  • ...and 3 more figures