Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Eye-in-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO

Zhenran Tang, Ruixuan Liu, Changliu Liu

TL;DR

Eye-in-Finger (EiF) addresses occlusion and precision challenges in manipulating small LEGO parts by embedding a low-cost endoscope camera directly at the end-effector. The approach combines pose estimation from partially occluded targets and tilt sensing via LED reflections to enable real-time visual servoing and robust error correction, substantially reducing calibration sensitivity. Key contributions include a compact EOAT with an integrated camera, a YOLOv8-seg–based knob detection pipeline with geometric circle fitting, and a reflection-based tilt estimator, validated by substantial improvements over wrist-mounted cameras (e.g., up to $8.8\times$ in position accuracy, $2.0$ mm calibration tolerance). The work demonstrates that end-effector–level perception can markedly improve accuracy, robustness, and teleoperation efficiency in delicate assembly tasks, with implications for broader use in precision robotics and automation.

Abstract

Manipulation and insertion of small and tight-toleranced objects in robotic assembly remain a critical challenge for vision-based robotics systems due to the required precision and cluttered environment. Conventional global or wrist-mounted cameras often suffer from occlusions when either assembling or disassembling from an existing structure. To address the challenge, this paper introduces "Eye-in-Finger", a novel tool design approach that enhances robotic manipulation by embedding low-cost, high-resolution perception directly at the tool tip. We validate our approach using LEGO assembly and disassembly tasks, which require the robot to manipulate in a cluttered environment and achieve sub-millimeter accuracy and robust error correction due to the tight tolerances. Experimental results demonstrate that our proposed system enables real-time, fine corrections to alignment error, increasing the tolerance of calibration error from 0.4mm to up to 2.0mm for the LEGO manipulation robot.

Eye-in-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO

TL;DR

Eye-in-Finger (EiF) addresses occlusion and precision challenges in manipulating small LEGO parts by embedding a low-cost endoscope camera directly at the end-effector. The approach combines pose estimation from partially occluded targets and tilt sensing via LED reflections to enable real-time visual servoing and robust error correction, substantially reducing calibration sensitivity. Key contributions include a compact EOAT with an integrated camera, a YOLOv8-seg–based knob detection pipeline with geometric circle fitting, and a reflection-based tilt estimator, validated by substantial improvements over wrist-mounted cameras (e.g., up to in position accuracy, mm calibration tolerance). The work demonstrates that end-effector–level perception can markedly improve accuracy, robustness, and teleoperation efficiency in delicate assembly tasks, with implications for broader use in precision robotics and automation.

Abstract

Manipulation and insertion of small and tight-toleranced objects in robotic assembly remain a critical challenge for vision-based robotics systems due to the required precision and cluttered environment. Conventional global or wrist-mounted cameras often suffer from occlusions when either assembling or disassembling from an existing structure. To address the challenge, this paper introduces "Eye-in-Finger", a novel tool design approach that enhances robotic manipulation by embedding low-cost, high-resolution perception directly at the tool tip. We validate our approach using LEGO assembly and disassembly tasks, which require the robot to manipulate in a cluttered environment and achieve sub-millimeter accuracy and robust error correction due to the tight tolerances. Experimental results demonstrate that our proposed system enables real-time, fine corrections to alignment error, increasing the tolerance of calibration error from 0.4mm to up to 2.0mm for the LEGO manipulation robot.

Paper Structure

This paper contains 17 sections, 2 equations, 7 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Eye-in-hand Visual Servoing
  5. Tactile Feedback
  6. LEGO Manipulation
  7. Methodology
  8. Eye-in-Finger Hardware Design
  9. Pose Estimation from Partially Occluded LEGO Target
  10. Tilt Estimation with Reflection
  11. LEGO Manipulation Process
  12. Experiments
  13. Measurement Accuracy
  14. Calibration Error Correction
  15. Eye-in-Finger Aided Teleoperation
  16. ...and 2 more sections

Figures (7)

  • Figure 1: (a): Top: EiF on a robot, holding a LEGO brick. Bottom: Cut-out views of EiF with key dimensions in millimeters. (b): A robot using EiF to correct misalignment and pick up LEGO bricks from a static board (top) and a moving hand (bottom). (c): Challenges with conventional approaches in LEGO manipulation that EiF addresses.
  • Figure 2: Illustration of the pipeline for pose estimation from partially occluded LEGO target.
  • Figure 3: Illustration of tilt estimation with reflection.
  • Figure 4: Measurement Accuracy
  • Figure 5: Comparison of the success rates between Lego manipulation with and without Eye-in-Finger under different magnitudes of calibration errors. 12 trials were conducted for each method at each magnitude
  • ...and 2 more figures