Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees

Yuhui Wang, Zhixiong Yang, Ming Zhang, Shihan Dou, Zhiheng Xi, Enyu Zhou, Senjie Jin, Yujiong Shen, Dingwei Zhu, Yi Dong, Tao Gui, Qi Zhang, Xuanjing Huang

Abstract

In the maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions. To enable fault trees stored as images to be directly processed by large language models, which can assist in tracking and analyzing malfunctions, we propose a novel textual representation of fault trees. Building on it, we construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments, evaluating a model's ability to assist in malfunction localization, which contains $3130$ entries and $40.75$ turns per entry on average. We train an end-to-end model to generate vague information to reflect user behavior and introduce long-range rollback and recovery procedures to simulate user error scenarios, enabling assessment of a model's integrated capabilities in task tracking and error recovery, and Gemini 2.5 pro archives the best performance.

JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees

Abstract

In the maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions. To enable fault trees stored as images to be directly processed by large language models, which can assist in tracking and analyzing malfunctions, we propose a novel textual representation of fault trees. Building on it, we construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments, evaluating a model's ability to assist in malfunction localization, which contains entries and turns per entry on average. We train an end-to-end model to generate vague information to reflect user behavior and introduce long-range rollback and recovery procedures to simulate user error scenarios, enabling assessment of a model's integrated capabilities in task tracking and error recovery, and Gemini 2.5 pro archives the best performance.
Paper Structure (42 sections, 3 figures, 10 tables, 1 algorithm)

This paper contains 42 sections, 3 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fault Tree Analysis
  4. Evaluation of Multi-Turn Dialogue Systems
  5. Evaluation of Agent Capabilities
  6. The JFTA Representation
  7. Syntax Definition
  8. Hierarchical Structure and References
  9. JFTA-Bench
  10. Fault Tree Collection
  11. Path Extraction and Dataset Construction
  12. Error Simulation and Recovery.
  13. Interaction Framework
  14. Dataset Statistics
  15. Experiments
  16. ...and 27 more sections

Figures (3)

  • Figure 1: The left panel illustrates the Human-in-the-loop data collection process. The middle panel depicts the path sampling procedure capable of handling DAG structures. The right panel demonstrates the multi-turn evaluation framework featuring user rollback scenarios.
  • Figure 2: Accuracy comparison between JFTA and Node-Edge representations across different difficulty levels and overall performance.
  • Figure 3: Performance of naturalness and fuzziness across different model sizes.