Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach

Weikun Peng, Jun Lv, Yuwei Zeng, Haonan Chen, Siheng Zhao, Jichen Sun, Cewu Lu, Lin Shao

TL;DR

This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie, and introduces the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video.

Abstract

The tie-knotting task is highly challenging due to the tie's high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline applies learned policy to real-world execution. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found https://tiebots.github.io/.

TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach

TL;DR

This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie, and introduces the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video.

Abstract

The tie-knotting task is highly challenging due to the tie's high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline applies learned policy to real-world execution. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found https://tiebots.github.io/.
Paper Structure (44 sections, 2 equations, 19 figures, 5 tables)

This paper contains 44 sections, 2 equations, 19 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cloth Manipulation
  4. Learning from Visual Demonstration
  5. Cloth State Estimation
  6. Technical Approach
  7. Real2Sim
  8. Learn@Sim
  9. Experiments
  10. Comparing TieBot and Baseline
  11. Evaluating Hierarchical Feature Matching (HFM)
  12. Apply HFM on Other Cloth Manipulation Tasks
  13. Conclusion
  14. More Discussion on Real2Sim
  15. Why not Use Video Tracking?
  16. ...and 29 more sections

Figures (19)

  • Figure 1: Our proposed TieBot performs a tie-knotting task. We leverage cloth simulation to recover the cloth's state from human demonstration and learn a goal-condition policy to accomplish the tie-knotting task.
  • Figure 2: TieBot utilizes simulation to estimate the tie's meshes from the demonstrated video. Then, using mesh sequences as subgoals, we introduce how to generate the robot's actions to manipulate the tie. The pipeline finally executes learned policy in real world.
  • Figure 3: Local Feature matching between two images. A hand caused a gap along the length of the tie during the demonstration.
  • Figure 4: The oriented keypoints to represent the state of the tie. The x,y,z axis are represented by the red, green, blue arrow, respectively.
  • Figure 5: The results of TieBot at different stages. We show different sides of the tie in red and blue and manipulation action in yellow to better visualize.
  • ...and 14 more figures