In-the-Wild Compliant Manipulation with UMI-FT

TL;DR

UMI-FT addresses the challenge of learning compliant manipulation by providing finger-level force/torque sensing via CoinFT sensors on each finger, enabling capture of both external contact and internal grasp forces during in-the-wild demonstrations. The authors train an Adaptive Compliance Policy that predicts target pose, grasp force, and stiffness, executed through a three-loop control architecture combining a learned policy with model-based admittance and grasp-force controllers. Across three contact-rich tasks—whiteboard wiping, zucchini skewering, and lightbulb insertion—UMI-FT with finger-level force sensing improves force regulation and task robustness, outperforming baselines lacking force feedback or compliance. The work demonstrates scalable data collection and learning for compliant manipulation, and the authors open-source hardware and software to facilitate broader adoption.

Abstract

Many manipulation tasks require careful force modulation. With insufficient force the task may fail, while excessive force could cause damage. The high cost, bulky size and fragility of commercial force/torque (F/T) sensors have limited large-scale, force-aware policy learning. We introduce UMI-FT, a handheld data-collection platform that mounts compact, six-axis force/torque sensors on each finger, enabling finger-level wrench measurements alongside RGB, depth, and pose. Using the multimodal data collected from this device, we train an adaptive compliance policy that predicts position targets, grasp force, and stiffness for execution on standard compliance controllers. In evaluations on three contact-rich, force-sensitive tasks (whiteboard wiping, skewering zucchini, and lightbulb insertion), UMI-FT enables policies that reliably regulate external contact forces and internal grasp forces, outperforming baselines that lack compliance or force sensing. UMI-FT offers a scalable path to learning compliant manipulation from in-the-wild demonstrations. We open-source the hardware and software to facilitate broader adoption at:https://umi-ft.github.io/.

Figures (8)

• Figure 1: UMI-FT design and capabilities. Left: UMI-FT uses coin-sized force/torque sensors to record the wrench at each compliant fingertip. Right: we evaluated UMI-FT in three tasks: lightbulb insertion, skewering zucchini, and whiteboard wiping. In each task, finger-level force sensing captures external contact forces and internal grasp forces, similar to forces experienced by a human operator during demonstrations. We use the external force measurement in a standard compliance controller to track a virtual spring target.
• Figure 2: UMI-FT finger calibration. (a) Raw capacitance data from CoinFT and the corresponding F/T from a reference sensor (Gamma, ATI) are collected with random F/T inputs. (b) The raw capacitance is mapped to F/T through an MLP layer. (c) Calibration results on unseen input. Only one of the shear axes (x, y) were plotted due to similarity.
• Figure 3: Controller structure of UMI-FT and the flow of information. Sensor modalities are listed on the left, with proprioception omitted for clarity. They flow to three controller loops in the middle, where the learned policy runs the slowest and generates reference targets to the other two model-based controllers.
• Figure 4: Structure of the adaptive compliance policy with modified inputs and outputs.
• Figure 5: Whiteboard Wiping policy rollout, test scenarios, and representative baseline failures.