Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AgentSGEN: Multi-Agent LLM in the Loop for Semantic Collaboration and GENeration of Synthetic Data

Vu Dinh Xuan, Hao Vo, David Murphy, Hoang D. Nguyen

TL;DR

AgentSGEN tackles data scarcity for safety-critical scenarios by proposing a dual-agent in-the-loop framework that separates semantic planning (Evaluator) from execution (Editor). The system relies on a structured Scene Graph and multi-stage context enrichment, enabling constraint-driven, fine-grained scene edits while maintaining visual realism. Extensive human and automatic evaluations on 53 indoor scenes demonstrate superior alignment with safety goals when collision checking is enabled, compared to Holodeck baselines. The approach offers a scalable, interpretable pathway toward robust synthetic data generation for construction safety and related high-stakes applications.

Abstract

The scarcity of data depicting dangerous situations presents a major obstacle to training AI systems for safety-critical applications, such as construction safety, where ethical and logistical barriers hinder real-world data collection. This creates an urgent need for an end-to-end framework to generate synthetic data that can bridge this gap. While existing methods can produce synthetic scenes, they often lack the semantic depth required for scene simulations, limiting their effectiveness. To address this, we propose a novel multi-agent framework that employs an iterative, in-the-loop collaboration between two agents: an Evaluator Agent, acting as an LLM-based judge to enforce semantic consistency and safety-specific constraints, and an Editor Agent, which generates and refines scenes based on this guidance. Powered by LLM's capabilities to reasoning and common-sense knowledge, this collaborative design produces synthetic images tailored to safety-critical scenarios. Our experiments suggest this design can generate useful scenes based on realistic specifications that address the shortcomings of prior approaches, balancing safety requirements with visual semantics. This iterative process holds promise for delivering robust, aesthetically sound simulations, offering a potential solution to the data scarcity challenge in multimedia safety applications.

AgentSGEN: Multi-Agent LLM in the Loop for Semantic Collaboration and GENeration of Synthetic Data

TL;DR

AgentSGEN tackles data scarcity for safety-critical scenarios by proposing a dual-agent in-the-loop framework that separates semantic planning (Evaluator) from execution (Editor). The system relies on a structured Scene Graph and multi-stage context enrichment, enabling constraint-driven, fine-grained scene edits while maintaining visual realism. Extensive human and automatic evaluations on 53 indoor scenes demonstrate superior alignment with safety goals when collision checking is enabled, compared to Holodeck baselines. The approach offers a scalable, interpretable pathway toward robust synthetic data generation for construction safety and related high-stakes applications.

Abstract

The scarcity of data depicting dangerous situations presents a major obstacle to training AI systems for safety-critical applications, such as construction safety, where ethical and logistical barriers hinder real-world data collection. This creates an urgent need for an end-to-end framework to generate synthetic data that can bridge this gap. While existing methods can produce synthetic scenes, they often lack the semantic depth required for scene simulations, limiting their effectiveness. To address this, we propose a novel multi-agent framework that employs an iterative, in-the-loop collaboration between two agents: an Evaluator Agent, acting as an LLM-based judge to enforce semantic consistency and safety-specific constraints, and an Editor Agent, which generates and refines scenes based on this guidance. Powered by LLM's capabilities to reasoning and common-sense knowledge, this collaborative design produces synthetic images tailored to safety-critical scenarios. Our experiments suggest this design can generate useful scenes based on realistic specifications that address the shortcomings of prior approaches, balancing safety requirements with visual semantics. This iterative process holds promise for delivering robust, aesthetically sound simulations, offering a potential solution to the data scarcity challenge in multimedia safety applications.
Paper Structure (20 sections, 1 equation, 10 figures)

This paper contains 20 sections, 1 equation, 10 figures.

Table of Contents

  1. Related Work
  2. Synthetic data generation
  3. Multi-Agent Systems
  4. Method
  5. Problem Formulation
  6. Stage 0: Context Enrichment and Constraint Generation
  7. Stage 1: Semantic Planning via Evaluator Agent (System 2)
  8. Stage 2: Interactive Scene Editing via Editor Agent (System 1)
  9. Scene Graph as Interactive Environment
  10. Limitations of Holodeck and Motivation
  11. Dual-Agent Cognitive Design
  12. Termination, Constraint Validation, and Feedback Loop
  13. Final Rendering and Synthetic Dataset Generation
  14. Evaluation
  15. Evaluation Protocol and Annotator Setup
  16. ...and 5 more sections

Figures (10)

  • Figure 1: Comparison of 3D scene generation methods.
  • Figure 2: Initial scene from Holodeck with corresponding scene graph and SGRender views.
  • Figure 3: Multi-modal context components passed to both agents. The context bundle consists of (i) goal-specific constraint requirements, (ii) symbolic scene graph, and (iii) 2D/3D renderings from SGRender.
  • Figure 4: AgentSGEN architecture showcasing semantic planning by the Evaluator Agent and iterative scene editing by the Editor Agent.
  • Figure 5: Examples of Holodeck's Limitation
  • ...and 5 more figures