Toward Autonomous Laboratory Safety Monitoring with Vision Language Models: Learning to See Hazards Through Scene Structure

TL;DR

This work tackles autonomous laboratory safety monitoring by leveraging vision–language models (VLMs) and a structured data generation pipeline that converts textual hazard scenarios into aligned scene graphs and photorealistic images. It demonstrates that VLMs reason effectively when provided with explicit scene structure but struggle with raw visual inputs, motivating a scene-graph–guided alignment (SG-G) that translates visuals into structured representations before hazard inference. A synthetic dataset of 1,207 aligned image–scene-graph–ground-truth triples across 362 scenarios is created from LabSafetyBench and used to evaluate seven open- and closed-source VLMs, revealing that textual scene graphs yield the strongest hazard detection performance while vision-only performance remains limited. The SG-G approach improves visual-only hazard detection by enforcing intermediate scene representations, and the results point to future work in instruction tuning to internalize scene-graph generation and safety reasoning for more reliable autonomous lab safety monitoring.

Abstract

Laboratories are prone to severe injuries from minor unsafe actions, yet continuous safety monitoring -- beyond mandatory pre-lab safety training -- is limited by human availability. Vision language models (VLMs) offer promise for autonomous laboratory safety monitoring, but their effectiveness in realistic settings is unclear due to the lack of visual evaluation data, as most safety incidents are documented primarily as unstructured text. To address this gap, we first introduce a structured data generation pipeline that converts textual laboratory scenarios into aligned triples of (image, scene graph, ground truth), using large language models as scene graph architects and image generation models as renderers. Our experiments on the synthetic dataset of 1,207 samples across 362 unique scenarios and seven open- and closed-source models show that VLMs perform effectively given textual scene graph, but degrade substantially in visual-only settings indicating difficulty in extracting structured object relationships directly from pixels. To overcome this, we propose a post-training context-engineering approach, scene-graph-guided alignment, to bridge perceptual gaps in VLMs by translating visual inputs into structured scene graphs better aligned with VLM reasoning, improving hazard detection performance in visual only settings.

Figures (20)

• Figure 1: Overview of our approach. (A) Structured Data Generation: Textual laboratory scenarios are first translated into scene graphs by an LLM acting as an architect, which are then used to condition a text-to-image model acting as a renderer to synthesize photorealistic laboratory images. (B) VLM as Hazard Detectors: In the visual-only setting, VLMs often fail to identify hazards directly from raw images, whereas asking VLMs to first infer a structured scene representation from the image enables safety reasoning and leads to correct hazard detection.
• Figure 2: End-to-end pipeline for dataset construction and hazard detection settings. (Left.) Textual laboratory scenarios are converted into structured scene graphs and ground-truth hazard annotations, which are then used to synthesize and align photorealistic laboratory images, yielding aligned triples through human and VLM-as-judge verification. (Right.) VLMs are evaluated under text-only, vision+text, and vision-only inputs. With our post-training context-engineering approach, scene-graph–guided reasoning, where VLMs first reconstruct a structured scene graph from visual input before performing hazard inference, improve performance for visual-only settings.
• Figure 3: Representative examples from our dataset illustrating hazardous and non-hazardous laboratory scenes.
• Figure 4: Distribution of hazardous and non-hazardous samples across laboratory subject categories. (Left.) Original scenario distribution with our ground-truth re-extraction in LabSafetyBench. (Right.) Distribution of our constructed image dataset after human and VLM-as-judge filtering. Our image dataset largely preserves the subject-wise and hazard-wise distribution of the original one.
• Figure 5: $F_1$ score comparison across hazard detection settings. Textual scene graph inputs without hazard attributes (TSG–H) achieve the best performance for all models, while vision-only (V) performs worst. Adding textual scene graphs to visual input (V+TSG–H) improves performance, and our scene-graph–guided (SG-G) approach further boosts visual-only hazard detection.