PREVENT: Proactive Risk Evaluation and Vigilant Execution of Tasks for Mobile Robotic Chemists using Multi-Modal Behavior Trees

TL;DR

This work tackles the safety and efficiency gap in autonomous mobile robotic chemists by introducing PREVENT, a multimodal hazard-aware framework that integrates two novel skills, Coordinated Inspection Navigation ($$CIN$$) and Inspection Before Manipulation ($$IBM$$), within a hierarchical Behavior Tree architecture. PREVENT fuses vision (CNN and Vision-Language Models) with olfactory sensing to continuously monitor navigation paths and inspect manipulation targets, guided by a decision rule $$a^* = f(a_i \mid x_1, x_2, x_3)$$ where $x_1 \in \{0,1\}$, $x_2 \in \mathbb{Z}$, and $x_3 \in \mathcal{L}$$, and further decomposes into $$a^{n*} = f_n(a_i \mid x_1^n, x_2^n, x_3^n)$$ and $$a^{m*} = f_m(a_i \mid x_1^m, x_2^m, x_3^m)$$ for navigation and manipulation respectively. The system is designed for seamless integration with existing architectures via APIs and a KUKA Sunrise.OS driver, and is evaluated on a PXRD workflow showing high deployment accuracy and complete avoidance of false positives/negatives when using the proposed multi-modal configuration. Key findings include that the fine-tuned vision modalities achieve near-perfect hazard detection across tasks (e.g., $>90\%$ accuracy, with some tasks approaching $98\%$), the olfactory modality provides critical context but alone is insufficient in all cases, and the multi-modal combination significantly outperforms uni- and bi-modal variants. Overall, PREVENT demonstrates safer autonomous operation, reduced need for human intervention, and practical integration potential, with future work aimed at extending hazard-centric generalization to more complex lab tasks.

Abstract

Mobile robotic chemists are a fast growing trend in the field of chemistry and materials research. However, so far these mobile robots lack workflow awareness skills. This poses the risk that even a small anomaly, such as an improperly capped sample vial could disrupt the entire workflow. This wastes time, and resources, and could pose risks to human researchers, such as exposure to toxic materials. Existing perception mechanisms can be used to predict anomalies but they often generate excessive false positives. This may halt workflow execution unnecessarily, requiring researchers to intervene and to resume the workflow when no problem actually exists, negating the benefits of autonomous operation. To address this problem, we propose PREVENT a system comprising navigation and manipulation skills based on a multimodal Behavior Tree (BT) approach that can be integrated into existing software architectures with minimal modifications. Our approach involves a hierarchical perception mechanism that exploits AI techniques and sensory feedback through Dexterous Vision and Navigational Vision cameras and an IoT gas sensor module for execution-related decision-making. Experimental evaluations show that the proposed approach is comparatively efficient and completely avoids both false negatives and false positives when tested in simulated risk scenarios within our robotic chemistry workflow. The results also show that the proposed multi-modal perception skills achieved deployment accuracies that were higher than the average of the corresponding uni-modal skills, both for navigation and for manipulation.

Figures (8)

• Figure 1: Mobile Robotic Chemist equipped with vision and olfactory perception modalities. Components ⓐ and ⓑ denote the vision and olfactory sensory modules of the robot arm, respectively. Components ⓒ and ⓓ correspond to the vision and olfactory sensors of the mobile base. Components ⓔ and ⓕ represent the robot's deck and onboard computer, respectively.
• Figure 2: PREVENT system architecture that enables hazard mitigation by integrating perception, decision-making, execution, and human and robot interface modules.
• Figure 3: Behavior tree structures of both skills: (a) CIN skill and (b) IBM skill.
• Figure 4: Custom-made driver of the MRC showing (a) task assignment and (b) task execution.
• Figure 5: Photographs showing simulated hazards in the MRC workflow; (a) Vial & chemical spillage during task T1, leading to a vial and chemicals on the lab floor that is too small to be detected by LIDAR, (b) Improper vial drop in capping station during task T2, (c) Broken glass pieces in the Chemspeed station during task T3. The relevant hazard is highlighted by a red rectangle in each case.