Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FeasibleCap: Real-Time Embodiment Constraint Guidance for In-the-Wild Robot Demonstration Collection

Zi Yin, Fanhong Li, Yun Gui, Jia Liu

TL;DR

Simulation experiments indicate that enforcing executability constraints during collection does not sacrifice cross-embodiment transfer across robot platforms, and simulation experiments further indicate that enforcing executability constraints during collection does not sacrifice cross-embodiment transfer across robot platforms.

Abstract

Gripper-in-hand data collection decouples demonstration acquisition from robot hardware, but whether a trajectory is executable on the target robot remains unknown until a separate replay-and-validate stage. Failed demonstrations therefore inflate the effective cost per usable trajectory through repeated collection, diagnosis, and validation. Existing collection-time feedback systems mitigate this issue but rely on head-worn AR/VR displays, robot-in-the-loop hardware, or learned dynamics models; real-time executability feedback has not yet been integrated into the gripper-in-hand data collection paradigm. We present \textbf{FeasibleCap}, a gripper-in-hand data collection system that brings real-time executability guidance into robot-free capture. At each frame, FeasibleCap checks reachability, joint-rate limits, and collisions against a target robot model and closes the loop through on-device visual overlays and haptic cues, allowing demonstrators to correct motions during collection without learned models, headsets, or robot hardware. On pick-and-place and tossing tasks, FeasibleCap improves replay success and reduces the fraction of infeasible frames, with the largest gains on tossing. Simulation experiments further indicate that enforcing executability constraints during collection does not sacrifice cross-embodiment transfer across robot platforms. Hardware designs and software are available at https://github.com/aod321/FeasibleCap.

FeasibleCap: Real-Time Embodiment Constraint Guidance for In-the-Wild Robot Demonstration Collection

TL;DR

Simulation experiments indicate that enforcing executability constraints during collection does not sacrifice cross-embodiment transfer across robot platforms, and simulation experiments further indicate that enforcing executability constraints during collection does not sacrifice cross-embodiment transfer across robot platforms.

Abstract

Gripper-in-hand data collection decouples demonstration acquisition from robot hardware, but whether a trajectory is executable on the target robot remains unknown until a separate replay-and-validate stage. Failed demonstrations therefore inflate the effective cost per usable trajectory through repeated collection, diagnosis, and validation. Existing collection-time feedback systems mitigate this issue but rely on head-worn AR/VR displays, robot-in-the-loop hardware, or learned dynamics models; real-time executability feedback has not yet been integrated into the gripper-in-hand data collection paradigm. We present \textbf{FeasibleCap}, a gripper-in-hand data collection system that brings real-time executability guidance into robot-free capture. At each frame, FeasibleCap checks reachability, joint-rate limits, and collisions against a target robot model and closes the loop through on-device visual overlays and haptic cues, allowing demonstrators to correct motions during collection without learned models, headsets, or robot hardware. On pick-and-place and tossing tasks, FeasibleCap improves replay success and reduces the fraction of infeasible frames, with the largest gains on tossing. Simulation experiments further indicate that enforcing executability constraints during collection does not sacrifice cross-embodiment transfer across robot platforms. Hardware designs and software are available at https://github.com/aod321/FeasibleCap.
Paper Structure (34 sections, 4 figures, 3 tables, 1 algorithm)

This paper contains 34 sections, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Handheld and Robot-Free Demonstration Collection
  4. Collection-Time Feedback for Demonstration Quality
  5. Inference-Time Embodiment Adaptation
  6. Method
  7. Problem Formulation
  8. System Overview
  9. Real-Time Feasibility Guidance
  10. Camera-to-TCP calibration.
  11. Clutch mechanism.
  12. Virtual robot base placement.
  13. Per-frame feasibility pipeline.
  14. Latency and throughput.
  15. Data Collection and Replay
  16. ...and 19 more sections

Figures (4)

  • Figure 1: FeasibleCap overview. An iPhone is mounted on a handheld gripper, providing real-time feasibility feedback via an on-screen indicator. The indicator turns red as the end-effector approaches workspace boundaries or joint-rate limits, guiding demonstrators to stay within the target robot's executable region.
  • Figure 2: Open-loop vs. closed-loop demonstration collection.Top: without guidance, infeasibility is discovered only at replay time, requiring costly re-collection. Bottom: FeasibleCap evaluates embodiment constraints in real time and feeds back visual and haptic cues, enabling the demonstrator to correct motions on the fly.
  • Figure 3: FeasibleCap system architecture.Top (Record): the iPhone processes each ARKit frame along two parallel paths---Path A streams compressed images and poses to the Raspberry Pi for synchronized multi-sensor recording in MCAP format; Path B runs the on-device feasibility pipeline (IK $\to$ FK $\to$ self-collision $\to$ feasibility check) and closes the feedback loop via AR ghost rendering and haptic vibration. The green arrow denotes the real-time closed-loop guidance, the core contribution of this work. Bottom (Replay): the iPhone triggers playback; the Raspberry Pi reads the MCAP trajectory and sends control commands to the target robot arm.
  • Figure 4: Per-frame feasibility timelines for representative trials. Each bar represents one trajectory; frames are colored green (feasible), yellow (warning), or red (infeasible). (a) Baseline, pick-and-place: 56% infeasible. (b) FeasibleCap, pick-and-place: 0% infeasible. (c) Baseline, tossing: 53% infeasible. (d) FeasibleCap, tossing: 14% infeasible.