No Need to Look! Locating and Grasping Objects by a Robot Arm Covered with Sensitive Skin

TL;DR

This work explored an extreme case of searching for and grasping objects in complete absence of visual input, relying on haptic feedback only, using the main novelty lies in the use of contacts over the complete surface of a robot manipulator covered with sensitive skin.

Abstract

Locating and grasping of objects by robots is typically performed using visual sensors. Haptic feedback from contacts with the environment is only secondary if present at all. In this work, we explored an extreme case of searching for and grasping objects in complete absence of visual input, relying on haptic feedback only. The main novelty lies in the use of contacts over the complete surface of a robot manipulator covered with sensitive skin. The search is divided into two phases: (1) coarse workspace exploration with the complete robot surface, followed by (2) precise localization using the end-effector equipped with a force/torque sensor. We systematically evaluated this method in simulation and on the real robot, demonstrating that diverse objects can be located, grasped, and put in a basket. The overall success rate on the real robot for one object was 85.7% with failures mainly while grasping specific objects. The method using whole-body contacts is six times faster compared to a baseline that uses haptic feedback only on the end-effector. We also show locating and grasping multiple objects on the table. This method is not restricted to our specific setup and can be deployed on any platform with the ability of sensing contacts over the entire body surface. This work holds promise for diverse applications in areas with challenging visual perception (due to lighting, dust, smoke, occlusion) such as in agriculture when fruits or vegetables need to be located inside foliage and picked.

Figures (9)

• Figure 1: Rough scan with the whole-body skin (top), precise localization with end-effector (bottom-left) and pre-grasp position (bottom-right).
• Figure 2: Robot with objects in the real world (left) and in simulation (right).
• Figure 3: Schema of the exploration pipeline. The robot starts at the rightmost position with the arm fully stretched forward and starts moving sideways with a fixed step. If it is not at the end position, it moves downwards at each step. In case of a collision, precise localization begins, followed by a grasping operation. After the grasping ends, or the robot moves to the lowest position, it returns upwards. The loop continues until the robot reaches the end position.
• Figure 4: (a) Top-view on table (orange), object (red) and projected area of the pad in contact (blue). $S_n, G_n$ are n-th start and end positions of the precise localization. (b) Top-view of the object (red) with both contacts $\mathbf{p}_1, \mathbf{p}_2$ (yellow) with approach vectors from both initial ($S, \mathbf{v}_1$) and orthogonal ($S_o, \mathbf{v}_2$) direction. (c) Top-view on the initial grasp pose. The colored coordinate frame shows directions for grasp poses refinement.
• Figure 5: Success rate for cylindrical object (red object in \ref{['fig:setup']}) in simulation at different positions in the workspace. Each square represents one position of the cylinder as its center. The values are computed over 5 repetitions at every location. The base of the robot is located at position (0, 0).