Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LeTac-MPC: Learning Model Predictive Control for Tactile-reactive Grasping

Zhengtong Xu, Yu She

TL;DR

This article introduces LeTac-MPC, a learning-based model predictive control for tactile-reactive grasping that incorporates a differentiable MPC layer designed to model the embeddings extracted by a neural network from tactile feedback.

Abstract

Grasping is a crucial task in robotics, necessitating tactile feedback and reactive grasping adjustments for robust grasping of objects under various conditions and with differing physical properties. In this paper, we introduce LeTac-MPC, a learning-based model predictive control (MPC) for tactile-reactive grasping. Our approach enables the gripper to grasp objects with different physical properties on dynamic and force-interactive tasks. We utilize a vision-based tactile sensor, GelSight, which is capable of perceiving high-resolution tactile feedback that contains information on the physical properties and states of the grasped object. LeTac-MPC incorporates a differentiable MPC layer designed to model the embeddings extracted by a neural network (NN) from tactile feedback. This design facilitates convergent and robust grasping control at a frequency of 25 Hz. We propose a fully automated data collection pipeline and collect a dataset only using standardized blocks with different physical properties. However, our trained controller can generalize to daily objects with different sizes, shapes, materials, and textures. The experimental results demonstrate the effectiveness and robustness of the proposed approach. We compare LeTac-MPC with two purely model-based tactile-reactive controllers (MPC and PD) and open-loop grasping. Our results show that LeTac-MPC has optimal performance in dynamic and force-interactive tasks and optimal generalizability. We release our code and dataset at https://github.com/ZhengtongXu/LeTac-MPC.

LeTac-MPC: Learning Model Predictive Control for Tactile-reactive Grasping

TL;DR

This article introduces LeTac-MPC, a learning-based model predictive control for tactile-reactive grasping that incorporates a differentiable MPC layer designed to model the embeddings extracted by a neural network from tactile feedback.

Abstract

Grasping is a crucial task in robotics, necessitating tactile feedback and reactive grasping adjustments for robust grasping of objects under various conditions and with differing physical properties. In this paper, we introduce LeTac-MPC, a learning-based model predictive control (MPC) for tactile-reactive grasping. Our approach enables the gripper to grasp objects with different physical properties on dynamic and force-interactive tasks. We utilize a vision-based tactile sensor, GelSight, which is capable of perceiving high-resolution tactile feedback that contains information on the physical properties and states of the grasped object. LeTac-MPC incorporates a differentiable MPC layer designed to model the embeddings extracted by a neural network (NN) from tactile feedback. This design facilitates convergent and robust grasping control at a frequency of 25 Hz. We propose a fully automated data collection pipeline and collect a dataset only using standardized blocks with different physical properties. However, our trained controller can generalize to daily objects with different sizes, shapes, materials, and textures. The experimental results demonstrate the effectiveness and robustness of the proposed approach. We compare LeTac-MPC with two purely model-based tactile-reactive controllers (MPC and PD) and open-loop grasping. Our results show that LeTac-MPC has optimal performance in dynamic and force-interactive tasks and optimal generalizability. We release our code and dataset at https://github.com/ZhengtongXu/LeTac-MPC.
Paper Structure (23 sections, 1 theorem, 12 equations, 18 figures, 5 tables)

This paper contains 23 sections, 1 theorem, 12 equations, 18 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Tactile Control
  4. Tactile Grasping
  5. Learning Physical Properties for Manipulation
  6. Approach
  7. Tactile Information Encoding
  8. Differentiable MPC Layer
  9. Training
  10. Data Collection
  11. Loss Function
  12. Controller Implementation
  13. Baseline Methods
  14. Proportional Derivative Control
  15. Model Predictive Control
  16. ...and 8 more sections

Key Result

Theorem 1

If $Q_v,Q_a,P > 0$ and $\mathbf{Q}_f$ is symmetric positive definite, then the resulting QP from equations eq:mpc_layer and eq:mpc_layer_tran is feasible regardless of any changes to the dimension of the embedding $M$, the prediction horizon $N$, and $\mathbf{A}_f$.

Figures (18)

  • Figure 1: LeTac-MPC network model. We use the raw image of tactile feedback as the model input. $\ell$ is our proposed loss function.
  • Figure 2: For each type of material, bottom left is the raw tactile image overlapping with the marker tracking when grasping the corresponding block and bottom right is the raw tactile image without marker tracking. These tactile images are collected under the same grasping force. Our goal here is to show that different materials have different physical properties, which leads to different features in the tactile image under the same grasping conditions. Therefore, we do not estimate the specific values of tangential force here.
  • Figure 3: Left: data collection setup. Right: automated data collection pipeline. The $xyz$-coordinate of the end-effector's position increments is shown in the left figure.
  • Figure 4: Collected data visualization.
  • Figure 5: Illustration of learned controller implementation. We implement the trained model as a real-time MPC controller. The proposed MPC layer combines model-based and data-driven formulations, allowing for the fine-tuning of the model-based part to achieve better control performance with real-time tactile feedback.
  • ...and 13 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof