Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Dynamic and High-Precision Method for Scenario-Based HRA Synthetic Data Collection in Multi-Agent Collaborative Environments Driven by LLMs

Xingyu Xiao, Peng Chen, Qianqian Jia, Jiejuan Tong, Jingang Liang, Haitao Wang

TL;DR

This work tackles the scarcity and rigidity of traditional HRA data by introducing WELLA, a dynamic workload estimation framework that leverages fine-tuned large language models trained on real HTGR operation data to simulate multi-agent cognitive load in scenario-driven settings. By collecting real-world operator data, generating virtual cognitive trajectories with macrocognitive theory, and applying supervised fine-tuning to a Qwen2.5-7B base model via Llama-factory, WELLA produces highly accurate workload predictions across multiple roles (RO1–RO3, CO, SO) and scenarios, outperforming commercial LLMs. The approach reduces reliance on expert input, enables real-time, scalable data generation, and enhances safety training and operational planning in complex, multi-agent nuclear environments. The results indicate significant potential for broader application in domains requiring dynamic workload assessment and scenario-based HRA data generation.

Abstract

HRA (Human Reliability Analysis) data is crucial for advancing HRA methodologies. however, existing data collection methods lack the necessary granularity, and most approaches fail to capture dynamic features. Additionally, many methods require expert knowledge as input, making them time-consuming and labor-intensive. To address these challenges, we propose a new paradigm for the automated collection of HRA data. Our approach focuses on key indicators behind human error, specifically measuring workload in collaborative settings. This study introduces a novel, scenario-driven method for workload estimation, leveraging fine-tuned large language models (LLMs). By training LLMs on real-world operational data from high-temperature gas-cooled reactors (HTGRs), we simulate human behavior and cognitive load in real time across various collaborative scenarios. The method dynamically adapts to changes in operator workload, providing more accurate, flexible, and scalable workload estimates. The results demonstrate that the proposed WELLA (Workload Estimation with LLMs and Agents) outperforms existing commercial LLM-based methods in terms of prediction accuracy.

A Dynamic and High-Precision Method for Scenario-Based HRA Synthetic Data Collection in Multi-Agent Collaborative Environments Driven by LLMs

TL;DR

This work tackles the scarcity and rigidity of traditional HRA data by introducing WELLA, a dynamic workload estimation framework that leverages fine-tuned large language models trained on real HTGR operation data to simulate multi-agent cognitive load in scenario-driven settings. By collecting real-world operator data, generating virtual cognitive trajectories with macrocognitive theory, and applying supervised fine-tuning to a Qwen2.5-7B base model via Llama-factory, WELLA produces highly accurate workload predictions across multiple roles (RO1–RO3, CO, SO) and scenarios, outperforming commercial LLMs. The approach reduces reliance on expert input, enables real-time, scalable data generation, and enhances safety training and operational planning in complex, multi-agent nuclear environments. The results indicate significant potential for broader application in domains requiring dynamic workload assessment and scenario-based HRA data generation.

Abstract

HRA (Human Reliability Analysis) data is crucial for advancing HRA methodologies. however, existing data collection methods lack the necessary granularity, and most approaches fail to capture dynamic features. Additionally, many methods require expert knowledge as input, making them time-consuming and labor-intensive. To address these challenges, we propose a new paradigm for the automated collection of HRA data. Our approach focuses on key indicators behind human error, specifically measuring workload in collaborative settings. This study introduces a novel, scenario-driven method for workload estimation, leveraging fine-tuned large language models (LLMs). By training LLMs on real-world operational data from high-temperature gas-cooled reactors (HTGRs), we simulate human behavior and cognitive load in real time across various collaborative scenarios. The method dynamically adapts to changes in operator workload, providing more accurate, flexible, and scalable workload estimates. The results demonstrate that the proposed WELLA (Workload Estimation with LLMs and Agents) outperforms existing commercial LLM-based methods in terms of prediction accuracy.

Paper Structure

This paper contains 17 sections, 2 equations, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Underlying Factors Contributing to Human Error
  4. Workload Estimation Models
  5. Large Language Models in Simulating Human Behavior
  6. Methodology
  7. Framework of the study
  8. Real-World Data Collection
  9. Vitural cognitive trajectory Generation
  10. Language Model Fine-Tuning
  11. Scenario Analysis
  12. Experiments and Results
  13. Implementation Detail
  14. Workload Estimation
  15. Comparisons with Existing Methods
  16. ...and 2 more sections

Figures (7)

  • Figure 1: IDHEAS-ECA HRA Process
  • Figure 2: The High-Temperature Reactor Simulator Operating Test Bench
  • Figure 3: A simplified model of macrocognitive functions. chang2014sacada
  • Figure 4: WELLA: Input-Output Dynamics in Scenario Analysis
  • Figure 5: Performance Comparison of WELLA, GPT-4, GPT-4O, and Claude Across Multiple Scenarios
  • ...and 2 more figures