Vision and Tactile Robotic System to Grasp Litter in Outdoor Environments

TL;DR

This work tackles outdoor litter collection by integrating a vision-based detection and 3D localisation pipeline with tactile feedback to enable stable grasping. The approach combines a CNN-based object detector with 3D point-cloud grasp estimation (GeoGrasp) and two low-cost DIGIT tactile sensors for contact and slip detection, forming a closed-loop manipulation system on a UR5e mounted on a mobile platform. Datasets for vision and tactile perception are created and used to train and validate each module, with outdoor tests showing substantial real-world performance (CSR around 0.94 overall and 0.80 on the first attempt). The results demonstrate the practicality of vision–tactile robotics for litter collection in varying outdoor environments, highlighting the value of low-cost tactile sensing and modular perception pipelines for real-world deployment.

Abstract

The accumulation of litter is increasing in many places and is consequently becoming a problem that must be dealt with. In this paper, we present a manipulator robotic system to collect litter in outdoor environments. This system has three functionalities. Firstly, it uses colour images to detect and recognise litter comprising different materials. Secondly, depth data are combined with pixels of waste objects to compute a 3D location and segment three-dimensional point clouds of the litter items in the scene. The grasp in 3 Degrees of Freedom (DoFs) is then estimated for a robot arm with a gripper for the segmented cloud of each instance of waste. Finally, two tactile-based algorithms are implemented and then employed in order to provide the gripper with a sense of touch. This work uses two low-cost visual-based tactile sensors at the fingertips. One of them addresses the detection of contact (which is obtained from tactile images) between the gripper and solid waste, while another has been designed to detect slippage in order to prevent the objects grasped from falling. Our proposal was successfully tested by carrying out extensive experimentation with different objects varying in size, texture, geometry and materials in different outdoor environments (a tiled pavement, a surface of stone/soil, and grass). Our system achieved an average score of 94% for the detection and Collection Success Rate (CSR) as regards its overall performance, and of 80% for the collection of items of litter at the first attempt.

Figures (24)

• Figure 1: Home, navigation and detection poses of BLUE robot. (Left) UR5e pose when BLUE is in home pose. (Bottom-left) Navigation pose when BLUE is moving around. (Right) Detection pose when BLUE is near an item of litter
• Figure 2: Scheme of our tactile-visual system for robotic grasping. It is made up of two main parts: waste detection and recognition (green part) and tactile perception in order to manipulate the item of waste (blue part).
• Figure 3: Mask R-CNN architecture with an example of the input and output from the litter detection task
• Figure 4: YOLACT architecture with an example of the input and output from the litter detection task
• Figure 5: Scheme of our object detection and grasping points calculation process. Our NNs obtain the segmented mask from the RGB image, which is used to extract the object point cloud in order to calculate the grasping points with our new version of the GeoGrasp algorithm