LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories

TL;DR

A systematic gap is revealed between general-domain MCQ accuracy and Semi-open QA safety performance, with models exhibiting an average drop of 32.0% in professional laboratory scenarios, particularly in hazard interpretation and safety-aware planning, which underscores the urgent necessity for safety-centric reasoning frameworks to ensure reliable autonomous scientific experimentation in embodied laboratory contexts.

Abstract

Artificial intelligence is increasingly catalyzing scientific automation, with multimodal large language model (MLLM) agents evolving from lab assistants into self-driving lab operators. This transition imposes stringent safety requirements on laboratory environments, where fragile glassware, hazardous substances, and high-precision laboratory equipment render planning errors or misinterpreted risks potentially irreversible. However, the safety awareness and decision-making reliability of embodied agents in such high-stakes settings remain insufficiently defined and evaluated. To bridge this gap, we introduce LABSHIELD, a realistic multi-view benchmark designed to assess MLLMs in hazard identification and safety-critical reasoning. Grounded in U.S. Occupational Safety and Health Administration (OSHA) standards and the Globally Harmonized System (GHS), LABSHIELD establishes a rigorous safety taxonomy spanning 164 operational tasks with diverse manipulation complexities and risk profiles. We evaluate 20 proprietary models, 9 open-source models, and 3 embodied models under a dual-track evaluation framework. Our results reveal a systematic gap between general-domain MCQ accuracy and Semi-open QA safety performance, with models exhibiting an average drop of 32.0% in professional laboratory scenarios, particularly in hazard interpretation and safety-aware planning. These findings underscore the urgent necessity for safety-centric reasoning frameworks to ensure reliable autonomous scientific experimentation in embodied laboratory contexts. The full dataset will be released soon.

Figures (18)

• Figure 1: The LabShield Diagnostic Framework. Top: Safety-centric evaluation pipeline and performance landscape of leading MLLMs. Bottom: Multi-view data acquisition workflow using an ego-centric robotic platform to capture high-fidelity multimodal data in real-world safety-critical laboratory environments.
• Figure 2: An example of the LabShield safety evaluation pipeline. Both MCQs and semi-open evaluations are presented under a representative hazardous laboratory scenario, with correct and incorrect planning outcomes highlighted to illustrate safety-critical decision-making behavior.
• Figure 3: Dataset Statistics.Left: Cross-distribution of Safety ($S_0$--$S_3$) and Operational ($L_0$--$L_3$) levels. Middle: Breakdown of VQA annotations by cognitive category (Perception, Reasoning, Planning). Right: Top-5 Unsafe Factors and Hazard Patterns across the four experimental scenarios.
• Figure 4: Safety Performance Scales with Hazard Perception and Pattern Recognition.
• Figure 5: Effect of in-context examples.