Virtual-force Based Visual Servo for Multiple Peg-in-Hole Assembly with Tightly Coupled Multi-Manipulator

TL;DR

This work tackles peg-in-hole assembly for large-size parts using tightly coupled multi-manipulators, where traditional single-arm methods struggle to align distant pegs and holes. It introduces a collaborative visual servo framework that relies on monocular in-hand cameras, two neural networks (state classification and positioning), and a virtual-force mechanism to fuse multi-camera visual features into a cooperative controller. A synthetic dataset with varied hole appearances is used to improve feature extraction, and the method demonstrates near-perfect success ($ ext{success} ightarrow 100\%$) in dual-arm dual-hole experiments with a small clearance ($0.2$ mm), showing robustness to camera calibration errors. The approach reduces reliance on precise calibration and planning, enabling efficient large-part MPiH assembly with potential for dexterous and human-robot collaborative extensions.

Abstract

Multiple Peg-in-Hole (MPiH) assembly is one of the fundamental tasks in robotic assembly. In the MPiH tasks for large-size parts, it is challenging for a single manipulator to simultaneously align multiple distant pegs and holes, necessitating tightly coupled multi-manipulator systems. For such MPiH tasks using tightly coupled multiple manipulators, we propose a collaborative visual servo control framework that uses only the monocular in-hand cameras of each manipulator to reduce positioning errors. Initially, we train a state classification neural network and a positioning neural network. The former divides the states of the peg and hole in the image into three categories: obscured, separated, and overlapped, while the latter determines the position of the peg and hole in the image. Based on these findings, we propose a method to integrate the visual features of multiple manipulators using virtual forces, which can naturally combine with the cooperative controller of the multi-manipulator system. To generalize our approach to holes of different appearances, we varied the appearance of the holes during the dataset generation process. The results confirm that by considering the appearance of the holes, classification accuracy and positioning precision can be improved. Finally, the results show that our method achieves 100\% success rate in dual-manipulator dual peg-in-hole tasks with a clearance of 0.2 mm, while robust to camera calibration errors.

Figures (11)

• Figure 1: The workflow of the proposed method.
• Figure 2: The images of scenes with different hole appearances. From top left to bottom right: hole1 to hole14.
• Figure 3: Images of the flat hole scene and the real peg-in-hole scenes.
• Figure 4: Schematic diagram of the vectors in the paper. ${\Sigma _o}$ denotes the coordinate system of the center of mass of the object, ${\Sigma _e}$ denotes the world coordinate system. The superscript of each vector indicates the coordinate frame in which it is expressed, and if it is relative to the ${\Sigma _e}$, the superscript is omitted by default.
• Figure 5: The test results of the state classification network and the positioning network. (a) The positioning errors of the hole; (b) The positioning errors of the peg; (c) The accuracy of the state classification network (EfficientNetV2-S); (d) Comparison of the accuracy of classical classification networks.