Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Visual Servoing under Human Supervision for Assembly Tasks

Victor Nan Fernandez-Ayala, Jorge Silva, Meng Guo, Dimos V. Dimarogonas

TL;DR

The paper tackles reliable robotic assembly for construction by fusing eye-in-hand visual servoing with Control Barrier Functions to maintain fiducial-marker visibility during pick-and-place, and adding an eye-to-hand module for precise placement. It introduces robustness to camera-pose errors through an adaptive, robust CBF formulation and integrates a Human-In-The-Loop controller to enhance flexibility while preserving safety. Key contributions include a field-of-view based CBF framework, robustification against pose uncertainty, a practical HIL control scheme, and a ROS2/MoveIt2-based software architecture demonstrated on 6-DoF mobile manipulators with collaborative execution. The work advances safe human-robot collaboration and reliable autonomous assembly in construction settings, with potential impact on safety, efficiency, and adaptability in complex, one-off tasks.

Abstract

We propose a framework enabling mobile manipulators to reliably complete pick-and-place tasks for assembling structures from construction blocks. The picking uses an eye-in-hand visual servoing controller for object tracking with Control Barrier Functions (CBFs) to ensure fiducial markers in the blocks remain visible. An additional robot with an eye-to-hand setup ensures precise placement, critical for structural stability. We integrate human-in-the-loop capabilities for flexibility and fault correction and analyze robustness to camera pose errors, proposing adapted barrier functions to handle them. Lastly, experiments validate the framework on 6-DoF mobile arms.

Robust Visual Servoing under Human Supervision for Assembly Tasks

TL;DR

The paper tackles reliable robotic assembly for construction by fusing eye-in-hand visual servoing with Control Barrier Functions to maintain fiducial-marker visibility during pick-and-place, and adding an eye-to-hand module for precise placement. It introduces robustness to camera-pose errors through an adaptive, robust CBF formulation and integrates a Human-In-The-Loop controller to enhance flexibility while preserving safety. Key contributions include a field-of-view based CBF framework, robustification against pose uncertainty, a practical HIL control scheme, and a ROS2/MoveIt2-based software architecture demonstrated on 6-DoF mobile manipulators with collaborative execution. The work advances safe human-robot collaboration and reliable autonomous assembly in construction settings, with potential impact on safety, efficiency, and adaptability in complex, one-off tasks.

Abstract

We propose a framework enabling mobile manipulators to reliably complete pick-and-place tasks for assembling structures from construction blocks. The picking uses an eye-in-hand visual servoing controller for object tracking with Control Barrier Functions (CBFs) to ensure fiducial markers in the blocks remain visible. An additional robot with an eye-to-hand setup ensures precise placement, critical for structural stability. We integrate human-in-the-loop capabilities for flexibility and fault correction and analyze robustness to camera pose errors, proposing adapted barrier functions to handle them. Lastly, experiments validate the framework on 6-DoF mobile arms.

Paper Structure

This paper contains 18 sections, 5 theorems, 22 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation
  4. Pinhole camera model
  5. Visual servoing
  6. Control barrier functions
  7. Main results
  8. Field of view
  9. CBFs for continued pose estimation
  10. System dynamics of the CBFs
  11. Robustifying the system
  12. Human-In-the-Loop (HIL) control
  13. Experimental Results
  14. Software architecture
  15. Results
  16. ...and 3 more sections

Key Result

Lemma 1

If $\prescript{\mathcal{F}_t}{}{}\boldsymbol{R}_{\hat{\mathcal{C}}_t} = I$, where $I$ is the identity matrix, and then $\mathcal{V}_{\Tilde{\mathcal{C}}_t}(\boldsymbol{K}, \boldsymbol{C}) \subseteq \mathcal{V}_{\mathcal{F}_t}(\boldsymbol{K}, \boldsymbol{C})$.

Figures (4)

  • Figure 1: Frames $\mathcal{E}_t$, $\hat{\mathcal{C}}_t$, $\mathcal{C}_t$ and image plane (left). Dotted lines show known transformations and distances. Field of view and ArUco marker (right).
  • Figure 2: Information flow diagram of the developed actions.
  • Figure 3: Experimental setup for assembly tasks using the (left) Hebi Rosie mobile manipulators and the (right) cubes with ArUco markers.
  • Figure 4: Time evolution of the CBFs with a grey line indicating zero value.

Theorems & Definitions (14)

  • Definition 1: Extended class $\mathcal{K}$ function
  • Definition 2: CBF
  • Remark 1
  • Lemma 1
  • proof
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • ...and 4 more