Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaze Detection and Analysis for Initiating Joint Activity in Industrial Human-Robot Collaboration

Pooja Prajod, Matteo Lavit Nicora, Marta Mondellini, Giovanni Tauro, Rocco Vertechy, Matteo Malosio, Elisabeth André

TL;DR

The paper investigates whether natural gaze toward a cobot can cue the initiation of joint activity in industrial human-robot collaboration. It uses a wizard-of-oz design with 37 participants performing an assembly task, employing a VGG16-based gaze classifier trained on ETH-XGaze to label gaze as cobot-directed, table-directed, or elsewhere, and NOVA annotations to mark joint-activity onset. Results show that approximately 84.88% of joint actions were preceded by gazing at the cobot, with gaze closely associated to the joint-activity window and relatively few unexpected gazes (mean around 9.7%). The findings support gaze as a social cue for initiating collaboration and motivate real-time, gaze-aware cobot adaptation to improve operator experience in industrial settings.

Abstract

Collaborative robots (cobots) are widely used in industrial applications, yet extensive research is still needed to enhance human-robot collaborations and operator experience. A potential approach to improve the collaboration experience involves adapting cobot behavior based on natural cues from the operator. Inspired by the literature on human-human interactions, we conducted a wizard-of-oz study to examine whether a gaze towards the cobot can serve as a trigger for initiating joint activities in collaborative sessions. In this study, 37 participants engaged in an assembly task while their gaze behavior was analyzed. We employ a gaze-based attention recognition model to identify when the participants look at the cobot. Our results indicate that in most cases (84.88\%), the joint activity is preceded by a gaze towards the cobot. Furthermore, during the entire assembly cycle, the participants tend to look at the cobot around the time of the joint activity. To the best of our knowledge, this is the first study to analyze the natural gaze behavior of participants working on a joint activity with a robot during a collaborative assembly task.

Gaze Detection and Analysis for Initiating Joint Activity in Industrial Human-Robot Collaboration

TL;DR

The paper investigates whether natural gaze toward a cobot can cue the initiation of joint activity in industrial human-robot collaboration. It uses a wizard-of-oz design with 37 participants performing an assembly task, employing a VGG16-based gaze classifier trained on ETH-XGaze to label gaze as cobot-directed, table-directed, or elsewhere, and NOVA annotations to mark joint-activity onset. Results show that approximately 84.88% of joint actions were preceded by gazing at the cobot, with gaze closely associated to the joint-activity window and relatively few unexpected gazes (mean around 9.7%). The findings support gaze as a social cue for initiating collaboration and motivate real-time, gaze-aware cobot adaptation to improve operator experience in industrial settings.

Abstract

Collaborative robots (cobots) are widely used in industrial applications, yet extensive research is still needed to enhance human-robot collaborations and operator experience. A potential approach to improve the collaboration experience involves adapting cobot behavior based on natural cues from the operator. Inspired by the literature on human-human interactions, we conducted a wizard-of-oz study to examine whether a gaze towards the cobot can serve as a trigger for initiating joint activities in collaborative sessions. In this study, 37 participants engaged in an assembly task while their gaze behavior was analyzed. We employ a gaze-based attention recognition model to identify when the participants look at the cobot. Our results indicate that in most cases (84.88\%), the joint activity is preceded by a gaze towards the cobot. Furthermore, during the entire assembly cycle, the participants tend to look at the cobot around the time of the joint activity. To the best of our knowledge, this is the first study to analyze the natural gaze behavior of participants working on a joint activity with a robot during a collaborative assembly task.
Paper Structure (14 sections, 6 figures)

This paper contains 14 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Background and Related works
  3. Gaze in Human-Human Interactions
  4. Gaze in Robotics
  5. Approach
  6. Experimental Setup
  7. Demographics and Assembly Task
  8. Ethical Approval
  9. Tools
  10. Attention Recognition Model
  11. NOVA Annotation
  12. Analysis
  13. Discussion
  14. Conclusions and Future works

Figures (6)

  • Figure 1: Schematic top-view of the experimental workcell.
  • Figure 2: Preassembled components for the cobot (Group A) and components assigned to the operator (Group B).
  • Figure 3: The images show the joint activity between the cobot and the participant and a no collaboration instance from two different viewpoints. On the top is an overview of the setup from the side. On the bottom is the front camera recordings that were used in the analysis.
  • Figure 4: A snapshot from the NOVA tool showing the predictions from the attention recognition model (top track), and the annotated joint activity start points (bottom track, red lines).
  • Figure 5: Box-plot visualization of pGazeJoint values computed from 37 participants
  • ...and 1 more figures