Relevance-driven Decision Making for Safer and More Efficient Human Robot Collaboration

TL;DR

The paper tackles safe and efficient human-robot collaboration in dynamic environments by introducing relevance as a context-aware dimensionality reduction that continuously perceives and interprets a scene. It proposes a two-loop asynchronous framework that combines real-time perception with LLM-derived world knowledge to quantify relevance and drive proactive decision-making, including a RAPF-based motion planner. Key contributions include a novel relevance quantification method predicting human objectives and relevant elements, the integration of LLMs into a real-time HRC loop, and a decision-making pipeline that significantly reduces collisions and collision frames compared with state-of-the-art, achieving objective and relevance prediction accuracies up to $0.90$ and $0.96$ and collision reductions of $63.76\%$ and $44.74\%$, respectively. The framework enables safer, more efficient HRC by guiding robot actions through relevance-informed perception and planning, illustrating a path toward tighter integration of LLM knowledge with real-time robotic control.

Abstract

Human brain possesses the ability to effectively focus on important environmental components, which enhances perception, learning, reasoning, and decision-making. Inspired by this cognitive mechanism, we introduced a novel concept termed relevance for Human-Robot Collaboration (HRC). Relevance is a dimensionality reduction process that incorporates a continuously operating perception module, evaluates cue sufficiency within the scene, and applies a flexible formulation and computation framework. In this paper, we present an enhanced two-loop framework that integrates real-time and asynchronous processing to quantify relevance and leverage it for safer and more efficient human-robot collaboration (HRC). The two-loop framework integrates an asynchronous loop, which leverages LLM world knowledge to quantify relevance, and a real-time loop, which performs scene understanding, human intent prediction, and decision-making based on relevance. HRC decision-making is enhanced by a relevance-based task allocation method, as well as a motion generation and collision avoidance approach that incorporates human trajectory prediction. Simulations and experiments show that our methodology for relevance quantification can accurately and robustly predict the human objective and relevance, with an average accuracy of up to 0.90 for objective prediction and up to 0.96 for relevance prediction. Moreover, our motion generation methodology reduces collision cases by 63.76% and collision frames by 44.74% when compared with a state-of-the-art (SOTA) collision avoidance method. Our framework and methodologies, with relevance, guide the robot on how to best assist humans and generate safer and more efficient actions for HRC.

Figures (1)

• Figure 1: The overview of the methodology for relevance quantification and application of relevance for safer human-robot collaboration. The inputs to the algorithm pipeline are the stream images from the camera, which are processed with scene understanding algorithms in the AI toolkit. Based on the scene understanding, the human intent is classified and fed into the LLM within the AI toolkit in an asynchronous manner. With the LLM results, the relevance can be quantified, which informs the decision-making module to generate actions for the robot for safer HRC.