Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TEXterity: Tactile Extrinsic deXterity

Antonia Bronars, Sangwoon Kim, Parag Patre, Alberto Rodriguez

TL;DR

TEXterity addresses precise in-hand object manipulation under occlusion by integrating tactile sensing with proprioception in a closed-loop framework. It combines discrete pose estimation via Viterbi decoding with a continuous iSAM-based estimator-controller to continually refine object pose and generate motion plans, using environment priors and a friction-based forward model. The approach yields large reductions in pose estimation error over single-shot and discrete methods and demonstrates practical manipulation tasks, including insertion with tight clearances. These results highlight the potential of tactile extrinsic dexterity for assembly and tool-use scenarios where visual pose information is limited, while also pointing to the need for compliant policies to achieve sub-millimeter precision.

Abstract

We introduce a novel approach that combines tactile estimation and control for in-hand object manipulation. By integrating measurements from robot kinematics and an image-based tactile sensor, our framework estimates and tracks object pose while simultaneously generating motion plans to control the pose of a grasped object. This approach consists of a discrete pose estimator that uses the Viterbi decoding algorithm to find the most likely sequence of object poses in a coarsely discretized grid, and a continuous pose estimator-controller to refine the pose estimate and accurately manipulate the pose of the grasped object. Our method is tested on diverse objects and configurations, achieving desired manipulation objectives and outperforming single-shot methods in estimation accuracy. The proposed approach holds potential for tasks requiring precise manipulation in scenarios where visual perception is limited, laying the foundation for closed-loop behavior applications such as assembly and tool use. Please see supplementary videos for real-world demonstration at https://sites.google.com/view/texterity.

TEXterity: Tactile Extrinsic deXterity

TL;DR

TEXterity addresses precise in-hand object manipulation under occlusion by integrating tactile sensing with proprioception in a closed-loop framework. It combines discrete pose estimation via Viterbi decoding with a continuous iSAM-based estimator-controller to continually refine object pose and generate motion plans, using environment priors and a friction-based forward model. The approach yields large reductions in pose estimation error over single-shot and discrete methods and demonstrates practical manipulation tasks, including insertion with tight clearances. These results highlight the potential of tactile extrinsic dexterity for assembly and tool-use scenarios where visual pose information is limited, while also pointing to the need for compliant policies to achieve sub-millimeter precision.

Abstract

We introduce a novel approach that combines tactile estimation and control for in-hand object manipulation. By integrating measurements from robot kinematics and an image-based tactile sensor, our framework estimates and tracks object pose while simultaneously generating motion plans to control the pose of a grasped object. This approach consists of a discrete pose estimator that uses the Viterbi decoding algorithm to find the most likely sequence of object poses in a coarsely discretized grid, and a continuous pose estimator-controller to refine the pose estimate and accurately manipulate the pose of the grasped object. Our method is tested on diverse objects and configurations, achieving desired manipulation objectives and outperforming single-shot methods in estimation accuracy. The proposed approach holds potential for tasks requiring precise manipulation in scenarios where visual perception is limited, laying the foundation for closed-loop behavior applications such as assembly and tool use. Please see supplementary videos for real-world demonstration at https://sites.google.com/view/texterity.
Paper Structure (11 sections, 10 equations, 5 figures, 2 tables)

This paper contains 11 sections, 10 equations, 5 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. METHOD
  4. Problem Formulation
  5. Overview
  6. Discrete Pose Estimator
  7. Continuous Pose Estimator-Controller
  8. Experiments and Results
  9. Performance Across Various Goal Configurations
  10. Real-World Application: Insertion Task
  11. Conclusions

Figures (5)

  • Figure 1: An example task that requires tactile extrinsic dexterity. A proper grasp is essential when using an Allen key to apply sufficient torque while fastening a hex bolt. The proposed method utilizes tactile sensing on the robot's finger to localize the grasped object's pose and also regrasp the object in hand by pushing it against the floor - effectively leveraging extrinsic dexterity.
  • Figure 2: Overview of the Simultaneous Tactile Estimation and Control Framework.
  • Figure 3: Graph Architecture of the Simultaneous Tactile Estimator-Controller.
  • Figure 4: Test objects with example tactile images and contact patch reconstruction.
  • Figure 5: Demonstrations of four types of goal configurations: (a) Relative Orientation + Stationary Extrinsic Contact, (b) Relative Orientation/Translation + Stationary Extrinsic Contact, (c) Relative Orientation + Global Orientation + Stationary Extrinsic Contact, and (d) Relative Orientation + Sliding Extrinsic Contact. The right column depicts normalized estimation accuracy for the proposed method and ablation models.