Integrating Large Language Models and Knowledge Graphs for Extraction and Validation of Textual Test Data

TL;DR

The paper addresses the challenge of extracting and validating highly heterogeneous test data from unstructured aerospace documents by proposing a hybrid architecture that combines Knowledge Graphs with Large Language Models. It extends the Semantic Sensor Network ontology for Test Reports, stores metadata in a KG, and exposes validated results through a Virtual Knowledge Graph accessed via SPARQL-to-SQL mappings, with LLM-based per-row compliance checking. A benchmarking study demonstrates that GPT-4 and Gemini Ultra excel at automated validation across different test types, supporting significant cost-benefit gains over manual processes. The work demonstrates practical impact by enabling cross-report analytics and lays out a path for broader deployment and future enhancements, such as automatic ontology construction and natural-language SPARQL querying.

Abstract

Aerospace manufacturing companies, such as Thales Alenia Space, design, develop, integrate, verify, and validate products characterized by high complexity and low volume. They carefully document all phases for each product but analyses across products are challenging due to the heterogeneity and unstructured nature of the data in documents. In this paper, we propose a hybrid methodology that leverages Knowledge Graphs (KGs) in conjunction with Large Language Models (LLMs) to extract and validate data contained in these documents. We consider a case study focused on test data related to electronic boards for satellites. To do so, we extend the Semantic Sensor Network ontology. We store the metadata of the reports in a KG, while the actual test results are stored in parquet accessible via a Virtual Knowledge Graph. The validation process is managed using an LLM-based approach. We also conduct a benchmarking study to evaluate the performance of state-of-the-art LLMs in executing this task. Finally, we analyze the costs and benefits of automating preexisting processes of manual data extraction and validation for subsequent cross-report analyses.

Figures (10)

• Figure 1: A portion of a color-coded spreadsheet that visually represents the heterogeneity within Test Reports, which typically contain around 23 sections. Green denotes uniform sections, while yellow represents variable ones. White cells indicate the absence of a section. Titles are intentionally obscured to protect confidential information.
• Figure 2: Examples of test results tables that illustrate the challenges of syntactic (shown in green), structural (shown in yellow), and semantic (shown in purple) heterogeneity.
• Figure 2: An example of a Test Report modeled using an extension of the SSN ontology that contains valid data for the POLVoltage and InternalIsolation tests.
• Figure 3: The preexisting manual data processing workflow in which potentially anomalous reports are subjected to manual extraction, cleaning, and validation.
• Figure 3: The mapping for the POLVoltage Observation.