Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Management for Automobile Failure Analysis Using Graph RAG

Yuta Ojima, Hiroki Sakaji, Tadashi Nakamura, Hiroaki Sakata, Kazuya Seki, Yuu Teshigawara, Masami Yamashita, Kazuhiro Aoyama

TL;DR

The paper tackles the challenge of transferring failure-analysis knowledge in the automotive industry by adapting Graph RAG to work with existing Failure Knowledge Graphs. It introduces an IR-based Graph RAG pipeline that retrieves related terms, extracts one-hop sub-graphs, filters with an LLM, and reasons to produce answers, addressing limitations of SP-based Graph RAG on existing KGs. Evaluations on Isuzu clutch-related failure data show the IR-based approach delivers superior, more concise responses compared to SP-based Graph RAG and ChatGPT, and results improve when including document-derived sentences in prompts. While promising, the study notes limitations such as single-hop sub-graphs and a small dataset, pointing to future work in multi-hop reasoning, human evaluation, and agent-based interactive processes for real-world deployment.

Abstract

This paper presents a knowledge management system for automobile failure analysis using retrieval-augmented generation (RAG) with large language models (LLMs) and knowledge graphs (KGs). In the automotive industry, there is a growing demand for knowledge transfer of failure analysis from experienced engineers to young engineers. However, failure events are phenomena that occur in a chain reaction, making them difficult for beginners to analyze them. While knowledge graphs, which can describe semantic relationships and structure information is effective in representing failure events, due to their capability of representing the relationships between components, there is much information in KGs, so it is challenging for young engineers to extract and understand sub-graphs from the KG. On the other hand, there is increasing interest in the use of Graph RAG, a type of RAG that combines LLMs and KGs for knowledge management. However, when using the current Graph RAG framework with an existing knowledge graph for automobile failures, several issues arise because it is difficult to generate executable queries for a knowledge graph database which is not constructed by LLMs. To address this, we focused on optimizing the Graph RAG pipeline for existing knowledge graphs. Using an original Q&A dataset, the ROUGE F1 score of the sentences generated by the proposed method showed an average improvement of 157.6% compared to the current method. This highlights the effectiveness of the proposed method for automobile failure analysis.

Knowledge Management for Automobile Failure Analysis Using Graph RAG

TL;DR

The paper tackles the challenge of transferring failure-analysis knowledge in the automotive industry by adapting Graph RAG to work with existing Failure Knowledge Graphs. It introduces an IR-based Graph RAG pipeline that retrieves related terms, extracts one-hop sub-graphs, filters with an LLM, and reasons to produce answers, addressing limitations of SP-based Graph RAG on existing KGs. Evaluations on Isuzu clutch-related failure data show the IR-based approach delivers superior, more concise responses compared to SP-based Graph RAG and ChatGPT, and results improve when including document-derived sentences in prompts. While promising, the study notes limitations such as single-hop sub-graphs and a small dataset, pointing to future work in multi-hop reasoning, human evaluation, and agent-based interactive processes for real-world deployment.

Abstract

This paper presents a knowledge management system for automobile failure analysis using retrieval-augmented generation (RAG) with large language models (LLMs) and knowledge graphs (KGs). In the automotive industry, there is a growing demand for knowledge transfer of failure analysis from experienced engineers to young engineers. However, failure events are phenomena that occur in a chain reaction, making them difficult for beginners to analyze them. While knowledge graphs, which can describe semantic relationships and structure information is effective in representing failure events, due to their capability of representing the relationships between components, there is much information in KGs, so it is challenging for young engineers to extract and understand sub-graphs from the KG. On the other hand, there is increasing interest in the use of Graph RAG, a type of RAG that combines LLMs and KGs for knowledge management. However, when using the current Graph RAG framework with an existing knowledge graph for automobile failures, several issues arise because it is difficult to generate executable queries for a knowledge graph database which is not constructed by LLMs. To address this, we focused on optimizing the Graph RAG pipeline for existing knowledge graphs. Using an original Q&A dataset, the ROUGE F1 score of the sentences generated by the proposed method showed an average improvement of 157.6% compared to the current method. This highlights the effectiveness of the proposed method for automobile failure analysis.

Paper Structure

This paper contains 18 sections, 5 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Failure Knowledge Graph
  4. Large Language Model and RAG
  5. Graph RAG
  6. Methodology
  7. Problem Definition
  8. Proposed Method
  9. Evaluation
  10. Experiment
  11. Experiment Setting
  12. Dataset
  13. Results
  14. Discussion
  15. Cause Analysis about Missing Information
  16. ...and 3 more sections

Figures (6)

  • Figure 1: Failure KGhara. This figure illustrates the nodes related to the clutch node. The original graph was created in Japanese, and the displayed excerpts are translated into English. In this representation, the blue labels represent the states of the system, while green labels denote the components involved.
  • Figure 2: Overview of RAG
  • Figure 3: Overview of the Graph RAG
  • Figure 4: Overview of the IR-based Graph RAG
  • Figure 5: Examples of sub-graph representations
  • ...and 1 more figures