Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scene Graph-Guided Generative AI Framework for Synthesizing and Evaluating Industrial Hazard Scenarios

Sanjay Acharjee, Abir Khan Ratul, Diego Patino, Md Nazmus Sakib

TL;DR

The paper addresses the lack of visual data for workplace hazard understanding by deriving photorealistic hazard scenes from OSHA accident narratives using a scene graph guided diffusion pipeline. It introduces a complete framework that uses GPT-4o to extract hazard rationales, semantic clustering to archetypes, LLaMA based scene graph modeling, and a graph aligned VQA evaluation to assess fidelity. The work's key contributions are the scene graph intermediate representation, the VQA Graph Score that outperforms CLIP/BLIP in semantic fidelity, and a scalable dataset with generation and evaluation tools. The results demonstrate that graph guided synthetic hazard data can support proactive safety modeling, enabling better generalization, simulation based training, and cross-domain data augmentation.

Abstract

Training vision models to detect workplace hazards accurately requires realistic images of unsafe conditions that could lead to accidents. However, acquiring such datasets is difficult because capturing accident-triggering scenarios as they occur is nearly impossible. To overcome this limitation, this study presents a novel scene graph-guided generative AI framework that synthesizes photorealistic images of hazardous scenarios grounded in historical Occupational Safety and Health Administration (OSHA) accident reports. OSHA narratives are analyzed using GPT-4o to extract structured hazard reasoning, which is converted into object-level scene graphs capturing spatial and contextual relationships essential for understanding risk. These graphs guide a text-to-image diffusion model to generate compositionally accurate hazard scenes. To evaluate the realism and semantic fidelity of the generated data, a visual question answering (VQA) framework is introduced. Across four state-of-the-art generative models, the proposed VQA Graph Score outperforms CLIP and BLIP metrics based on entropy-based validation, confirming its higher discriminative sensitivity.

Scene Graph-Guided Generative AI Framework for Synthesizing and Evaluating Industrial Hazard Scenarios

TL;DR

The paper addresses the lack of visual data for workplace hazard understanding by deriving photorealistic hazard scenes from OSHA accident narratives using a scene graph guided diffusion pipeline. It introduces a complete framework that uses GPT-4o to extract hazard rationales, semantic clustering to archetypes, LLaMA based scene graph modeling, and a graph aligned VQA evaluation to assess fidelity. The work's key contributions are the scene graph intermediate representation, the VQA Graph Score that outperforms CLIP/BLIP in semantic fidelity, and a scalable dataset with generation and evaluation tools. The results demonstrate that graph guided synthetic hazard data can support proactive safety modeling, enabling better generalization, simulation based training, and cross-domain data augmentation.

Abstract

Training vision models to detect workplace hazards accurately requires realistic images of unsafe conditions that could lead to accidents. However, acquiring such datasets is difficult because capturing accident-triggering scenarios as they occur is nearly impossible. To overcome this limitation, this study presents a novel scene graph-guided generative AI framework that synthesizes photorealistic images of hazardous scenarios grounded in historical Occupational Safety and Health Administration (OSHA) accident reports. OSHA narratives are analyzed using GPT-4o to extract structured hazard reasoning, which is converted into object-level scene graphs capturing spatial and contextual relationships essential for understanding risk. These graphs guide a text-to-image diffusion model to generate compositionally accurate hazard scenes. To evaluate the realism and semantic fidelity of the generated data, a visual question answering (VQA) framework is introduced. Across four state-of-the-art generative models, the proposed VQA Graph Score outperforms CLIP and BLIP metrics based on entropy-based validation, confirming its higher discriminative sensitivity.

Paper Structure

This paper contains 18 sections, 13 equations, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative AI for Hazard Synthesis
  4. Limitations in Evaluation Metrics
  5. Scene Graphs for Hazard Reasoning and Generation
  6. Methodology
  7. Methodology
  8. Classification and Clustering of OSHA Accident Narratives with GPT-4o Reasoning
  9. Semantic Embedding and Clustering of Hazard Rationales
  10. Semantic Embedding and Clustering of Hazard Rationales
  11. Density-Based Clustering Using HDBSCAN
  12. Selection of Representative Hazard Archetypes
  13. Scene Graph-Based Scenario Modeling
  14. Experimental Results
  15. Evaluation of State-of-the-Art Generative Models
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Structured prompt framework for hazard extraction.
  • Figure 2: t-SNE visualization of clustered hazard rationales.
  • Figure 3: Example scene graph for a slip hazard.
  • Figure 4: Pipeline for scene generation and evaluation.
  • Figure 5: Outputs from four generative models across hazard prompts.
  • ...and 2 more figures