Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GOFAI meets Generative AI: Development of Expert Systems by means of Large Language Models

Eduardo C. Garrido-Merchán, Cristina Puente

TL;DR

This paper presents a hybrid framework for building expert systems by extracting structured, symbolic knowledge from large language models (LLMs) and encoding it into Prolog for transparent, rule-based reasoning. By constraining the knowledge domain and using carefully designed prompts, the approach yields verifiable Prolog knowledge bases with high factual accuracy, validated both statistically and via Wikidata, while maintaining interpretability through a symbolic representation. The methodology combines the probabilistic capabilities of LLMs with the deterministic rigor of symbolic systems, supported by PAC-style guarantees and scalable expansion strategies. The work demonstrates strong factual alignment (over 99% in their tests) and demonstrates practical pathways for deploying dependable AI in sensitive domains, with future work on improved entity linking and cross-LLM information gain analysis.

Abstract

The development of large language models (LLMs) has successfully transformed knowledge-based systems such as open domain question nswering, which can automatically produce vast amounts of seemingly coherent information. Yet, those models have several disadvantages like hallucinations or confident generation of incorrect or unverifiable facts. In this paper, we introduce a new approach to the development of expert systems using LLMs in a controlled and transparent way. By limiting the domain and employing a well-structured prompt-based extraction approach, we produce a symbolic representation of knowledge in Prolog, which can be validated and corrected by human experts. This approach also guarantees interpretability, scalability and reliability of the developed expert systems. Via quantitative and qualitative experiments with Claude Sonnet 3.7 and GPT-4.1, we show strong adherence to facts and semantic coherence on our generated knowledge bases. We present a transparent hybrid solution that combines the recall capacity of LLMs with the precision of symbolic systems, thereby laying the foundation for dependable AI applications in sensitive domains.

GOFAI meets Generative AI: Development of Expert Systems by means of Large Language Models

TL;DR

This paper presents a hybrid framework for building expert systems by extracting structured, symbolic knowledge from large language models (LLMs) and encoding it into Prolog for transparent, rule-based reasoning. By constraining the knowledge domain and using carefully designed prompts, the approach yields verifiable Prolog knowledge bases with high factual accuracy, validated both statistically and via Wikidata, while maintaining interpretability through a symbolic representation. The methodology combines the probabilistic capabilities of LLMs with the deterministic rigor of symbolic systems, supported by PAC-style guarantees and scalable expansion strategies. The work demonstrates strong factual alignment (over 99% in their tests) and demonstrates practical pathways for deploying dependable AI in sensitive domains, with future work on improved entity linking and cross-LLM information gain analysis.

Abstract

The development of large language models (LLMs) has successfully transformed knowledge-based systems such as open domain question nswering, which can automatically produce vast amounts of seemingly coherent information. Yet, those models have several disadvantages like hallucinations or confident generation of incorrect or unverifiable facts. In this paper, we introduce a new approach to the development of expert systems using LLMs in a controlled and transparent way. By limiting the domain and employing a well-structured prompt-based extraction approach, we produce a symbolic representation of knowledge in Prolog, which can be validated and corrected by human experts. This approach also guarantees interpretability, scalability and reliability of the developed expert systems. Via quantitative and qualitative experiments with Claude Sonnet 3.7 and GPT-4.1, we show strong adherence to facts and semantic coherence on our generated knowledge bases. We present a transparent hybrid solution that combines the recall capacity of LLMs with the precision of symbolic systems, thereby laying the foundation for dependable AI applications in sensitive domains.

Paper Structure

This paper contains 17 sections, 6 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary and related work
  3. Knowledge adquisition
  4. Limitations of LLMs regarding verification systems and computability
  5. Methodology
  6. Theoretical Framework: LLMs as Probabilistic Knowledge Oracles
  7. Sample Complexity and PAC Guarantees
  8. Experimental section
  9. Factual Verification via Statistical Hypothesis Testing
  10. Qualitative Behavior Across Expansion Depths
  11. Automated Validation Against Wikidata
  12. Methodology
  13. Results
  14. Analysis by Predicate Type
  15. Discussion
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Knowledge visualization of 100 nodes of causal and domain relations extracted about the cosmic horror author H. P. Lovecraft from Claude 3.7 Sonnet.
  • Figure 2: Nodes can be pruned using the visualization script. We include in this figure 20 nodes of causal and domain relations of the same concept than in the previous figure
  • Figure 3: Diagram flow with all the processes involved in the creation of the expert system from the LLM.
  • Figure 4: Knowledge network of Plato of the LLM. 10 main relations. We can see famous concepts being developed of Plato like the cave allegory, the philosopher king, the republic, the theory of forms or the tripartite soul or that Plato has founded the academia.
  • Figure 5: Knowledge graph extracted from Grok regarding Plato.