Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM Agents Implement an NLG System from Scratch: Building Interpretable Rule-Based RDF-to-Text Generators

Mateusz Lango, Ondřej Dušek

TL;DR

This paper introduces a neurosymbolic framework where multiple LLM agents collectively design a rule-based NLG system in Python to verbalize RDF triples, eliminating the need for supervised in-domain data and enabling fast CPU-only inference. The approach yields an interpretable, maintainable generator that reduces hallucinations compared with neural baselines and remains competitive on standard metrics. Extensive experiments across WebNLG, OpenDialKG, and GEM 2024 show favorable performance, strong speedups, and solid human judgments, while ablations and interpretability studies validate the design choices. The work highlights a practical path to transparent, efficient RDF-to-text generation with potential for manual refinement and safe deployment.

Abstract

We present a novel neurosymbolic framework for RDF-to-text generation, in which the model is "trained" through collaborative interactions among multiple LLM agents rather than traditional backpropagation. The LLM agents produce rule-based Python code for a generator for the given domain, based on RDF triples only, with no in-domain human reference texts. The resulting system is fully interpretable, requires no supervised training data, and generates text nearly instantaneously using only a single CPU. Our experiments on the WebNLG and OpenDialKG data show that outputs produced by our approach reduce hallucination, with only slight fluency penalties compared to finetuned or prompted language models

LLM Agents Implement an NLG System from Scratch: Building Interpretable Rule-Based RDF-to-Text Generators

TL;DR

This paper introduces a neurosymbolic framework where multiple LLM agents collectively design a rule-based NLG system in Python to verbalize RDF triples, eliminating the need for supervised in-domain data and enabling fast CPU-only inference. The approach yields an interpretable, maintainable generator that reduces hallucinations compared with neural baselines and remains competitive on standard metrics. Extensive experiments across WebNLG, OpenDialKG, and GEM 2024 show favorable performance, strong speedups, and solid human judgments, while ablations and interpretability studies validate the design choices. The work highlights a practical path to transparent, efficient RDF-to-text generation with potential for manual refinement and safe deployment.

Abstract

We present a novel neurosymbolic framework for RDF-to-text generation, in which the model is "trained" through collaborative interactions among multiple LLM agents rather than traditional backpropagation. The LLM agents produce rule-based Python code for a generator for the given domain, based on RDF triples only, with no in-domain human reference texts. The resulting system is fully interpretable, requires no supervised training data, and generates text nearly instantaneously using only a single CPU. Our experiments on the WebNLG and OpenDialKG data show that outputs produced by our approach reduce hallucination, with only slight fluency penalties compared to finetuned or prompted language models

Paper Structure

This paper contains 29 sections, 11 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Training a rule-based NLG system
  3. Test Engineer
  4. Software Architect
  5. Software Engineer
  6. Evaluator
  7. Code Analyst
  8. Experiments
  9. Experimental setup
  10. Baselines
  11. Datasets
  12. Our approach
  13. Results of reference-based metrics
  14. Results of reference-less metrics
  15. Ablation experiments
  16. ...and 14 more sections

Figures (11)

  • Figure 1: Overview of the presented approach. LLM Agents (boxes with green border) interact with each other to write an entire NLG system in pure Python during the training phase. The final system is fully interpretable, easy to edit by a human, and does not need any LLM during inference.
  • Figure 2: Prompt of the Software Architect
  • Figure 3: Prompt of the Software Engineer
  • Figure 4: Prompt of the Evaluator
  • Figure 5: Prompt of the Code Analyst (Part 1/2, continued on the next page)
  • ...and 6 more figures