A versatile robotic hand with 3D perception, force sensing for autonomous manipulation

TL;DR

This work tackles the challenge of versatile autonomous manipulation by introducing a lightweight, manufacturable gripper with integrated 3D perception and force sensing, designed for research and education. The hardware relies on off-the-shelf components and 3D-printed PLA parts to deliver independent finger torque control, a palm-mounted depth camera, and a broad aperture for in-hand perception. The software stack combines YOLOv5-based segmentation, 3D point-cloud processing, and a task-and-motion planning framework using PDDL 2.1 and FastDownward, with continuous replanning guided by a Behavior Tree-based execution layer. Empirical results show force control up to $32\ \mathrm{N}$ with $0.08\ \mathrm{N}$ increments, sub-millimeter assembly performance on a Siemens gear task, and robust long-horizon manipulation through replanning, underscoring the platform's potential for education and rapid prototyping in household, industrial, and warehouse contexts.

Abstract

We describe a force-controlled robotic gripper with built-in tactile and 3D perception. We also describe a complete autonomous manipulation pipeline consisting of object detection, segmentation, point cloud processing, force-controlled manipulation, and symbolic (re)-planning. The design emphasizes versatility in terms of applications, manufacturability, use of commercial off-the-shelf parts, and open-source software. We validate the design by characterizing force control (achieving up to 32N, controllable in steps of 0.08N), force measurement, and two manipulation demonstrations: assembly of the Siemens gear assembly problem, and a sensor-based stacking task requiring replanning. These demonstrate robust execution of long sequences of sensor-based manipulation tasks, which makes the resulting platform a solid foundation for researchers in task-and-motion planning, educators, and quick prototyping of household, industrial and warehouse automation tasks.

Figures (6)

• Figure 1: All-in-one manipulation architecture that emphasizes manufacturability, active compliance, robust execution, and versatility. A light-weight software architecture combines object detection, segmentation, point cloud processing, with planning, inference and execution, creating a platform for research in task-and-motion planning for complex manipulation problems.
• Figure 2: Torque Diagram to Calculate Estimated Force on the Object.
• Figure 3: Left: CAD Drawings of the gripper from the top (A), bottom (B), and exploded view (C). Each motor independently actuates a finger, providing independent torque-based control. The camera is integrated into the palm. Right: Bill of Materials with approximated cost. PLA with a cost of $0.05 per $cm^2$.
• Figure 4: Gripper grasping the mustard container from the YCB dataset from an off-center position (left). Limiting torque during approach prevents the mustard from moving as the right finger makes contact. Dashed vertical lines indicate contact by the right finger, contact with both fingers, and opening (from left to right).
• Figure 5: Demonstration of accuracy and precision by reliably assembling the "Siemens gear assembly problem" vecerik2019practical with sub-millimeter accuracy requirements.