Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PHMForge: A Scenario-Driven Agentic Benchmark for Industrial Asset Lifecycle Maintenance

Ayan Das, Dhaval Patel

Abstract

Large language model (LLM) agents are increasingly deployed for complex tool-orchestration tasks, yet existing benchmarks fail to capture the rigorous demands of industrial domains where incorrect decisions carry significant safety and financial consequences. To address this critical gap, we introduce PHMForge, the first comprehensive benchmark specifically designed to evaluate LLM agents on Prognostics and Health Management (PHM) tasks through realistic interactions with domain-specific MCP servers. Our benchmark encompasses 75 expert-curated scenarios spanning 7 industrial asset classes (turbofan engines, bearings, electric motors, gearboxes, aero-engines) across 5 core task categories: Remaining Useful Life (RUL) Prediction, Fault Classification, Engine Health Analysis, Cost-Benefit Analysis, and Safety/Policy Evaluation. To enable rigorous evaluation, we construct 65 specialized tools across two MCP servers and implement execution-based evaluators with task-commensurate metrics: MAE/RMSE for regression, F1-score for classification, and categorical matching for health assessments. Through extensive evaluation of leading frameworks (ReAct, Cursor Agent, Claude Code) paired with frontier LLMs (Claude Sonnet 4.0, GPT-4o, Granite-3.0-8B), we find that even top-performing configurations achieve only 68\% task completion, with systematic failures in tool orchestration (23\% incorrect sequencing), multi-asset reasoning (14.9 percentage point degradation), and cross-equipment generalization (42.7\% on held-out datasets). We open-source our complete benchmark, including scenario specifications, ground truth templates, tool implementations, and evaluation scripts, to catalyze research in agentic industrial AI.

PHMForge: A Scenario-Driven Agentic Benchmark for Industrial Asset Lifecycle Maintenance

Abstract

Large language model (LLM) agents are increasingly deployed for complex tool-orchestration tasks, yet existing benchmarks fail to capture the rigorous demands of industrial domains where incorrect decisions carry significant safety and financial consequences. To address this critical gap, we introduce PHMForge, the first comprehensive benchmark specifically designed to evaluate LLM agents on Prognostics and Health Management (PHM) tasks through realistic interactions with domain-specific MCP servers. Our benchmark encompasses 75 expert-curated scenarios spanning 7 industrial asset classes (turbofan engines, bearings, electric motors, gearboxes, aero-engines) across 5 core task categories: Remaining Useful Life (RUL) Prediction, Fault Classification, Engine Health Analysis, Cost-Benefit Analysis, and Safety/Policy Evaluation. To enable rigorous evaluation, we construct 65 specialized tools across two MCP servers and implement execution-based evaluators with task-commensurate metrics: MAE/RMSE for regression, F1-score for classification, and categorical matching for health assessments. Through extensive evaluation of leading frameworks (ReAct, Cursor Agent, Claude Code) paired with frontier LLMs (Claude Sonnet 4.0, GPT-4o, Granite-3.0-8B), we find that even top-performing configurations achieve only 68\% task completion, with systematic failures in tool orchestration (23\% incorrect sequencing), multi-asset reasoning (14.9 percentage point degradation), and cross-equipment generalization (42.7\% on held-out datasets). We open-source our complete benchmark, including scenario specifications, ground truth templates, tool implementations, and evaluation scripts, to catalyze research in agentic industrial AI.

Paper Structure

This paper contains 24 sections, 1 equation, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work and Research Positioning
  3. PHM Benchmarks and Time-Series QA
  4. Language Agent Frameworks
  5. Agentic and Tool-Use Benchmarks
  6. Our Contributions in Context
  7. Scenario Design and Dataset Curation
  8. Scenario Categorization
  9. Dataset Selection and Scenario Curation
  10. Scenario Generation Pipeline : Human Filtering and Annotation
  11. Ground Truth Development, Utilization and Validation
  12. Evaluation Framework
  13. MCP Servers
  14. LLM Backbones and Agentic Frameworks
  15. Experimental Benchmark
  16. ...and 9 more sections

Figures (4)

  • Figure 1: PHMForge Architecture. Industrial stakeholder queries are processed by agentic frameworks and LLM backbones, which orchestrate tools across two domain-specific MCP servers: the Prognostics Server (38 tools for RUL, fault classification, and health analysis) and the Intelligent Maintenance Server (27 tools for cost-benefit and safety evaluation).
  • Figure 2: Process behind how we expanded our scenarios, choice of datasets and corresponding tasks associated with the datasets that made up the scenarios
  • Figure 3: Scenario Generation Pipeline
  • Figure 4: Ground truth structure for open-ended (RUL Prediction, Fault Classification) and close-ended (Engine Health Analysis) scenarios