RoTipBot: Robotic Handling of Thin and Flexible Objects using Rotatable Tactile Sensors

TL;DR

RoTipBot presents a novel deformable-object manipulation framework that fuses rotatable vision-based tactile sensing with a counting-then-grasping strategy to handle multiple thin layers in one closure. The RoTip sensor provides omnidirectional contact information, enabling continuous feeding and counting, while tactile-based adjustments ensure secure two-finger contact and robust grasping. Key results show average plane-normal estimation error of $1.51^{\circ}$ and up to $3\times$ faster operation than state-of-the-art methods, with the system capable of counting and grasping multiple layers simultaneously. These findings demonstrate the practical viability of mobilised tactile sensing for efficient, high-precision manipulation of deformable objects and open avenues for broader applications and future learning-based enhancements.

Abstract

This paper introduces RoTipBot, a novel robotic system for handling thin, flexible objects. Different from previous works that are limited to singulating them using suction cups or soft grippers, RoTipBot can count multiple layers and then grasp them simultaneously in a single grasp closure. Specifically, we first develop a vision-based tactile sensor named RoTip that can rotate and sense contact information around its tip. Equipped with two RoTip sensors, RoTipBot rolls and feeds multiple layers of thin, flexible objects into the centre between its fingers, enabling effective grasping. Moreover, we design a tactile-based grasping strategy that uses RoTip's sensing ability to ensure both fingers maintain secure contact with the object while accurately counting the number of fed objects. Extensive experiments demonstrate the efficacy of the RoTip sensor and the RoTipBot approach. The results show that RoTipBot not only achieves a higher success rate but also grasps and counts multiple layers simultaneously -- capabilities not possible with previous methods. Furthermore, RoTipBot operates up to three times faster than state-of-the-art methods. The success of RoTipBot paves the way for future research in object manipulation using mobilised tactile sensors. All the materials used in this paper are available at https://sites.google.com/view/rotipbot.

Figures (19)

• Figure 1: (a) A demonstration of RoTipBot. The tactile sensors ensure good contact with objects, while the rotation capability feeds multiple layers of thin, flexible objects into the centre for grasping and counting. Different transparencies of the paper represent states at different time steps. (b) Snapshots of the feeding process for multiple print papers. (c-e) Sketches comparing RoTipBot to approaches based on suction cups and soft grippers. RoTipBot can count multiple layers and then grasp them simultaneously in a single grasp closure, whereas the other methods cannot.
• Figure 2: From Left to Right:(a) Design overview of the RoTip sensor, and the exploded view of RoTip's three modules: (b) fixing module that serves as the structural backbone, ensuring stability and interconnection among sensor components; (c) transmission module that facilitates rotational movement, enabling dynamic functionality; and (d) finger body that constitutes the tactile interface, providing the tactile sensing capabilities.
• Figure 3: Hardware Setup and Coordinates. The hardware setup consists of a 6-DOF UR5e robotic arm, a Robotiq 2F-85 gripper, an Intel RealSense D435 RGB-D camera, and a laptop stand for holding the printer papers or other thin and flexible objects. The fingertips of the Robotiq gripper are replaced with two RoTip sensors. $W$ and $E$ represent the world coordinate that is located at the base of UR5e and the end-effector coordinate, respectively. $C$ and $M$ represent the camera coordinate and the marker coordinate, respectively. $T$ is the target coordinate related to the marker coordinate and used to direct the movement of the end-effector.
• Figure 4: An overview of our RoTipBot for thin and flexible handling. (a) Vision-Based Grasping Generation: Given an RGB-D image obtained from a camera, we generate the grasp proposal for guiding the robot to contact and grasp the object. (b) Tactile-based Adjustment for Two-finger Sufficient Contact: To compensate for the noise from visual perception, RoTip’s sensing capability is then utilised to adjust the robot’s end-effector. Once both RoTip sensors are in contact with the object, the end-effector will be rotated around its $x$-axis to an angle inclined to the object for feeding and grasping. (c) Continuous Adjustment and Tactile Counting: A continuous pose adjustment approach is proposed to ensure two-finger contact while feeding multiple thin and flexible objects. Tactile sensing is also employed to count the number of fed pages, a process referred to as tactile counting. (d) Object Grasping: Finally, the gripper is closed to pick up the objects.
• Figure 5: Pipeline for tactile-based thin object normal prediction. Given a tactile image as input, a fine-tuned DeepLab v3+ chen2018encoder network is used to segment the contact area of the RoTip sensor. Then, the contour of the contact area can be obtained based on the camera projection model and the finger geometry. Finally, a RANSAC-based plane fitting approach is used to derive the parameters of the contact plane.