An Extensive Evaluation of Factual Consistency in Large Language Models for Data-to-Text Generation

TL;DR

This work addresses the problem of factual consistency in Data-to-Text Generation (DTG) by performing an extensive evaluation across five DTG datasets and twelve large language models from five families, using four automatic metrics and human judgments. The authors formalize factual consistency and source-reference divergence, and employ parameter-efficient fine-tuning (QLoRA) with careful experimental controls to compare model size effects via Average Rate of Change (AROC). Key findings show that the Llama 2 family often delivers the strongest factual consistency, while larger model sizes generally improve consistency, though dataset properties modulate which smaller models can remain competitive; source-reference divergence consistently reduces factuality across models. The study provides practical guidance for deploying DTG systems, highlighting when larger models are advantageous and how divergence between source and reference data impacts reliability, with future work exploring prompting-based and other parameter-efficient fine-tuning strategies.

Abstract

Large Language Models (LLMs) have shown exceptional performance across various Data-to-Text Generation (DTG) tasks. However, generating factually consistent text in DTG remains challenging for LLMs. Despite this, in-depth evaluations of LLM factual consistency for DTG remain missing in the current literature. This paper addresses this gap by providing an extensive evaluation of factual consistency in LLMs for DTG. Our evaluation covers five widely used DTG datasets (E2E, ViGGo, WikiTableText, DART, and WebNLG) and five prominent LLM families (T5, BART, OPT, BLOOM, and Llama 2). To ensure a thorough evaluation of factual consistency, we use four state-of-the-art automatic metrics and include essential human assessments. Our extensive evaluations reveals three key findings regarding factual consistency in LLMs for DTG. First, Llama 2 often excels in generating factually consistent text, although smaller models like T5 and BART can achieve strong factual consistency on larger, lexically less-diverse datasets. Second, the average rate of change (AROC) indicates that increasing model size (number of model trainable parameters) generally enhances factual consistency of LLMs in DTG. Third, we observe that source-reference divergence (i.e., when the reference text diverges semantically from the source) typically reduces the factual consistency of LLMs in DTG.

Figures (14)

• Figure 1: The three primary types of Data-to-Text Generation (DTG): graph-to-text, meaning representation (MR)-to-text, and table-to-text.
• Figure 2: Two popular language model architectures: encoder-decoder (left) and decoder-only (right). The encoder processes input ($i$); the decoder generates output ($o$).
• Figure 3: An example of lack of factual consistency from the DART (graph-to-text) dataset, where the claim (generated text) introduces incorrect information—'that comes in the moderate price range'—not found in the source (reference) data.
• Figure 4: An illustration of source-reference divergence from E2E dusek2020evaluating dataset. The reference text includes an additional facts (bolded and underlined)---'is reasonably priced'---which in absent in the source data.
• Figure 5: All twelve LLMs across five widely used families (BART, T5, OPT, BLOOM, and Llama2) arranged by their model sizes.