Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparing the Consistency of User Studies Conducted in Simulations and Laboratory Settings

Jonathan Hümmer, Dominik Riedelbauch, Dominik Henrich

TL;DR

It is shown how a simulation environment can be aligned with a laboratory setting in which a robot and a human perform pick-and-place tasks together, and a user study indicates that participants' assembly decisions and perception of the situation are consistent across these different environments.

Abstract

Human-robot collaboration enables highly adaptive co-working. The variety of resulting workflows makes it difficult to measure metrics as, e.g. makespans or idle times for multiple systems and tasks in a comparable manner. This issue can be addressed with virtual commissioning, where arbitrary numbers of non-deterministic human-robot workflows in assembly tasks can be simulated. To this end, data-driven models of human decisions are needed. Gathering the required large corpus of data with on-site user studies is quite time-consuming. In comparison, simulation-based studies (e.g., by crowdsourcing) would allow us to access a large pool of study participants with less effort. To rely on respective study results, human action sequences observed in a browser-based simulation environment must be shown to match those gathered in a laboratory setting. To this end, this work aims to understand to what extent cooperative assembly work in a simulated environment differs from that in an on-site laboratory setting. We show how a simulation environment can be aligned with a laboratory setting in which a robot and a human perform pick-and-place tasks together. A user study (N=29) indicates that participants' assembly decisions and perception of the situation are consistent across these different environments.

Comparing the Consistency of User Studies Conducted in Simulations and Laboratory Settings

TL;DR

It is shown how a simulation environment can be aligned with a laboratory setting in which a robot and a human perform pick-and-place tasks together, and a user study indicates that participants' assembly decisions and perception of the situation are consistent across these different environments.

Abstract

Human-robot collaboration enables highly adaptive co-working. The variety of resulting workflows makes it difficult to measure metrics as, e.g. makespans or idle times for multiple systems and tasks in a comparable manner. This issue can be addressed with virtual commissioning, where arbitrary numbers of non-deterministic human-robot workflows in assembly tasks can be simulated. To this end, data-driven models of human decisions are needed. Gathering the required large corpus of data with on-site user studies is quite time-consuming. In comparison, simulation-based studies (e.g., by crowdsourcing) would allow us to access a large pool of study participants with less effort. To rely on respective study results, human action sequences observed in a browser-based simulation environment must be shown to match those gathered in a laboratory setting. To this end, this work aims to understand to what extent cooperative assembly work in a simulated environment differs from that in an on-site laboratory setting. We show how a simulation environment can be aligned with a laboratory setting in which a robot and a human perform pick-and-place tasks together. A user study (N=29) indicates that participants' assembly decisions and perception of the situation are consistent across these different environments.

Paper Structure

This paper contains 30 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. User Studies without Physical Robot Setups
  4. Human-Robot Interaction in Virtual Environments
  5. The question of consistent results
  6. Methods
  7. Alignment of Experimental Setup with the ACT-R Model
  8. Production memory
  9. Environment
  10. Declarative Memory
  11. Motion
  12. Visual Perception
  13. Auditory perception
  14. User Study Design
  15. Hypotheses
  16. ...and 15 more sections

Figures (5)

  • Figure 1: We compare participants' assembly strategies when solving pick and place tasks with a robot in aligned laboratory (left) and simulation (right) setups. Human speed is regulated by a "traffic light" (a), and identical images show the task (b).
  • Figure 2: We derive factors influencing human decision-making from the ACT-R model's most commonly used components (adapted from ritterACTRCognitiveArchitecture2019).
  • Figure 3: The descriptions of the main study tasks. To derive assembly sequences each action received a label. During the study, however, participants were shown unlabeled images.
  • Figure 4: The observed assembly sequences can be grouped into clusters (A, B, C, D, E) indicating aligned working strategies. Individual strategies were observed for participants in the Lab and the Sim condition alike.
  • Figure 5: Post-study questionnaire results