Gaze-based Human-Robot Interaction System for Infrastructure Inspections

TL;DR

This work tackles the limitations of qualitative visual infrastructure inspections by introducing a gaze-based human-robot interaction system powered by mixed reality. It processes inspector gaze to infer attention states (scanning, focusing, inspecting) and, during inspecting, uses fixation data to estimate defect size and compute a pose for a holographic MR drone to capture imagery. Experimental results from laboratory tests show accurate gaze positioning and line tracking, with promising defect quantification performance and consistent results across participants. The approach offers a non-intrusive method to augment expert inspection workflows, potentially improving speed, repeatability, and decision quality in infrastructure monitoring.

Abstract

Routine inspections for critical infrastructures such as bridges are required in most jurisdictions worldwide. Such routine inspections are largely visual in nature, which are qualitative, subjective, and not repeatable. Although robotic infrastructure inspections address such limitations, they cannot replace the superior ability of experts to make decisions in complex situations, thus making human-robot interaction systems a promising technology. This study presents a novel gaze-based human-robot interaction system, designed to augment the visual inspection performance through mixed reality. Through holograms from a mixed reality device, gaze can be utilized effectively to estimate the properties of the defect in real-time. Additionally, inspectors can monitor the inspection progress online, which enhances the speed of the entire inspection process. Limited controlled experiments demonstrate its effectiveness across various users and defect types. To our knowledge, this is the first demonstration of the real-time application of eye gaze in civil infrastructure inspections.

Figures (8)

• Figure 1: An example of gaze-based human-robot collaborative inspection through mixed reality.
• Figure 2: Architecture of the proposed gaze-based inspection system.
• Figure 3: Analysis of fixations collected during the inspecting step: (a) fixation collection until there is little or no change of the convex hull size of fixations; (b) defect properties evaluation; and (c) 5D pose estimation of the MR drone.
• Figure 4: Eye tracking accuracy test: (a) 9 points for gaze positioning test (b) 8 lines for line tracking test.
• Figure 5: Result of gaze accuracy evaluation; red dots are from the gaze positioning test, and blue dots are from the line tracking test.