Relevance for Human Robot Collaboration

TL;DR

This work defines relevance as a continuous dimensionality-reduction framework for human-robot collaboration (HRC) that concentrates processing on contextually important scene elements. It integrates a continuously running multi-modal perception module, an event-based trigger mechanism, and a probabilistic, hierarchical scene representation to compute relevance scores and identify relevant sets, enabling proactive, seamless assistance. In simulation, the framework achieves high metrics, e.g., precision $0.99$, recall $0.94$, F1 $0.96$, and object ratio $0.94$ under thresholds $\tau_c=\tau_e=0.2$, and it delivers substantial improvements in task planning time (up to $79.56\%$) and perception latency (up to $26.53\%$), while reducing human inquiries (up to $80.84\%$) and enhancing safety. A real-world demonstration with a UR5-based robot shows the system autonomously and adaptively assisting two humans in making coffee, illustrating proactive, human-centric assistance enabled by relevance.

Abstract

Inspired by the human ability to selectively focus on relevant information, this paper introduces relevance, a novel dimensionality reduction process for human-robot collaboration (HRC). Our approach incorporates a continuously operating perception module, evaluates cue sufficiency within the scene, and applies a flexible formulation and computation framework. To accurately and efficiently quantify relevance, we developed an event-based framework that maintains a continuous perception of the scene and selectively triggers relevance determination. Within this framework, we developed a probabilistic methodology, which considers various factors and is built on a novel structured scene representation. Simulation results demonstrate that the relevance framework and methodology accurately predict the relevance of a general HRC setup, achieving a precision of 0.99, a recall of 0.94, an F1 score of 0.96, and an object ratio of 0.94. Relevance can be broadly applied to several areas in HRC to accurately improve task planning time by 79.56% compared with pure planning for a cereal task, reduce perception latency by up to 26.53% for an object detector, improve HRC safety by up to 13.50% and reduce the number of inquiries for HRC by 80.84%. A real-world demonstration showcases the relevance framework's ability to intelligently and seamlessly assist humans in everyday tasks.

Figures (4)

• Figure 1: Overview of the framework for relevance quantification and application of relevance for proactive human-robot collaboration. The framework consists of a continuously running perception module, a trigger check module to selectively initialize relevance determination, a two-level relevance determination methodology, and a decision-making module to generate natural and efficient human robot interaction.
• Figure 2: Simulation results for the coffee domain. The values in the figures are averaged across 30 cases for each threshold combination. When $\tau_c$ and $\tau_e$ equal to 0.2, our methodology archives a recall $\mathfrak{R}$ of 0.94, a precision $\mathfrak{P}$ of 0.99, an F1 score $\mathfrak{F}$ of 0.96, and an object ratio $\mathfrak{N}$ of 0.94.
• Figure 3: Number of cases each element is predicted to be relevant out of 30 cases without considering constraints. $\tau_c$ and $\tau_e$ are 0.2. Our relevance quantification method predicts relevance accurately and reliably.
• Figure 4: The illustration of (a) experimental setup and (b-h) demonstration results. With relevance, the robot successfully and seamlessly assists two human agents with cold-brew coffee drinking.