GNN-based Decentralized Perception in Multirobot Systems for Predicting Worker Actions

TL;DR

A perception framework is introduced that enables mobile robots to understand and share information about human actions in a decentralized way and increases system resilience, making the multi-robot setup more reliable in dynamic industrial settings.

Abstract

In industrial environments, predicting human actions is essential for ensuring safe and effective collaboration between humans and robots. This paper introduces a perception framework that enables mobile robots to understand and share information about human actions in a decentralized way. The framework first allows each robot to build a spatial graph representing its surroundings, which it then shares with other robots. This shared spatial data is combined with temporal information to track human behavior over time. A swarm-inspired decision-making process is used to ensure all robots agree on a unified interpretation of the human's actions. Results show that adding more robots and incorporating longer time sequences improve prediction accuracy. Additionally, the consensus mechanism increases system resilience, making the multi-robot setup more reliable in dynamic industrial settings.

Figures (7)

• Figure 1: Overview of the multi-robot system deployed in Isaac Sim for human intent prediction. Robots generate spatial graph representations, integrate neighbor information, employ RNNs for temporal understanding, and converge using a swarm-intelligence-inspired consensus mechanism
• Figure 2: System architecture for multi-robot human intent prediction: a) robot detects the human and objects of interest in the scene, feature vectors are created through an encoder. b) Graph representation of the scene is created. c) Intent prediction based on GNN is made. This information is shared with other robots and the Temporal block. d) Human pose keypoints and spatial information from other robots are aggregated and passed through the temporal understanding block to make ego and collective intent predictions. This information is again shared with other robots. e) Concensus mechanism utilizes the predictions and quality of visual information from the robot to converge to a single decision about the intent.
• Figure 3: Composition of temporal feature vectors for GRU models. For the Ego-GRU, the feature vector combines the node embeddings from the ego robot and pose information obtained from the object detector. For the Collective-GRU, the feature vector integrates the node embeddings from both the ego robot and neighboring robots, along with the pose information.
• Figure 4: Performance comparison of Ego-GRU and Collective-GRU (with 3 robots) models using different temporal horizons: (a) 1s past observation predicting 1s into the future, (b) 2s past predicting 2s into the future, (c) 3s past predicting 3s into the future.
• Figure 5: : Effect of increasing the number of robots in our multi-robot system on prediction accuracy.