"Where is My Troubleshooting Procedure?": Studying the Potential of RAG in Assisting Failure Resolution of Large Cyber-Physical System

TL;DR

The paper investigates the feasibility of using Retrieval Augmented Generation (RAG) to aid operators in selecting and describing troubleshooting procedures from large, natural-language manuals for a naval cyber-physical system. It designs a fully local RAG pipeline that embeds, retrieves, and grounds candidate procedures while leveraging LLMs to generate operational steps, evaluated via a comprehensive, judge-based protocol across multiple questions and configurations. Results show that RAG can quickly propose plausible procedures and even derive undocumented steps in some cases, but robust cross-validation with explicit sources is essential due to risks of inaccuracies and hallucinations, especially under terminological ambiguity or incomplete documentation. The findings highlight a clear speed-accuracy trade-off across model sizes, underscore the importance of domain terminology, and point to practical implications for deploying safe, ground-truthed conversational troubleshooting tools in safety-critical CPS contexts.

Abstract

In today's complex industrial environments, operators must often navigate through extensive technical manuals to identify troubleshooting procedures that may help react to some observed failure symptoms. These manuals, written in natural language, describe many steps in detail. Unfortunately, the number, magnitude, and articulation of these descriptions can significantly slow down and complicate the retrieval of the correct procedure during critical incidents. Interestingly, Retrieval Augmented Generation (RAG) enables the development of tools based on conversational interfaces that can assist operators in their retrieval tasks, improving their capability to respond to incidents. This paper presents the results of a set of experiments that derive from the analysis of the troubleshooting procedures available in Fincantieri, a large international company developing complex naval cyber-physical systems. Results show that RAG can assist operators in reacting promptly to failure symptoms, although specific measures have to be taken into consideration to cross-validate recommendations before actuating them.

Figures (6)

• Figure 1: Architecture of the proposed RAG for the recommendation of troubleshooting procedures.
• Figure 2: Total number of questions submitted to the RAG in our evaluation.
• Figure 3: Boxplot representing the assessment of the answers to the accurate questions with context.
• Figure 4: Boxplot representing the assessment of the answers to inaccurate questions with context.
• Figure 5: Boxplot representing the assessment of the answers to questions without context.