Leveraging Large Language Models for Building Interpretable Rule-Based Data-to-Text Systems

TL;DR

The paper tackles data-to-text generation from RDF triples by merging interpretability with neural guidance: it trains a fully interpretable rule-based NLG system by using a large language model to generate executable Python rules. Rules are generated, tested, and corrected in a loop, with generalization boosted via predicate clustering and synthetic data. On WebNLG, the resulting system achieves fewer hallucinations and substantial CPU-efficiency, ranking competitively in text quality while offering strong interpretability and controllability. The approach trades some raw neural performance for transparency, speed, and ease of debugging, and lays groundwork for extending to out-of-domain predicates and more complex sentence-level constructs.

Abstract

We introduce a simple approach that uses a large language model (LLM) to automatically implement a fully interpretable rule-based data-to-text system in pure Python. Experimental evaluation on the WebNLG dataset showed that such a constructed system produces text of better quality (according to the BLEU and BLEURT metrics) than the same LLM prompted to directly produce outputs, and produces fewer hallucinations than a BART language model fine-tuned on the same data. Furthermore, at runtime, the approach generates text in a fraction of the processing time required by neural approaches, using only a single CPU

Figures (6)

• Figure 1: An overview of the training process of our rule-based system. Note that the output of the training process is a NLG system implemented in pure Python code that does not need access to the LLM to generate text.
• Figure 2: Prompt used to generate rules in our approach.
• Figure 3: Prompt used to inquire for rule edits in our approach.
• Figure 4: Prompt used to generate artificial training instances in our approach.
• Figure 5: Prompt for the zero-shot prompted LLM direct data-to-text generation baseline.