Factual Inconsistency in Data-to-Text Generation Scales Exponentially with LLM Size: A Statistical Validation

TL;DR

This study shows that factual inconsistency in data-to-text generation scales exponentially with LLM size rather than by the commonly assumed power law. It introduces a rigorous three-stage statistical framework to compare power-law and exponential scaling using five D2T datasets across three LLM families and four automated metrics. Through MLE with robust $\mathcal{L}_{\text{Huber}}$ loss and Vuong tests, the authors find exponential scaling provides a superior fit in most settings, with specific exceptions tied to datasets or model families. The findings offer practical guidance for selecting model sizes and evaluation strategies to enhance factual reliability in D2T systems, while highlighting limitations of relying solely on automated metrics and prompting further human-centric validation.

Abstract

Monitoring factual inconsistency is essential for ensuring trustworthiness in data-to-text generation (D2T). While large language models (LLMs) have demonstrated exceptional performance across various D2T tasks, previous studies on scaling laws have primarily focused on generalization error through power law scaling to LLM size (i.e., the number of model parameters). However, no research has examined the impact of LLM size on factual inconsistency in D2T. In this paper, we investigate how factual inconsistency in D2T scales with LLM size by exploring two scaling laws: power law and exponential scaling. To rigorously evaluate and compare these scaling laws, we employ a statistical validation framework consisting of three key stages: predictive performance estimation, goodness-of-fit assessment, and comparative analysis. For a comprehensive empirical study, we analyze three popular LLM families across five D2T datasets, measuring factual inconsistency inversely using four state-of-the-art consistency metrics. Our findings, based on exhaustive empirical results and validated through our framework, reveal that, contrary to the widely assumed power law scaling, factual inconsistency in D2T follows an exponential scaling with LLM size.

Figures (18)

• Figure 1: Example of data-to-text generation from the DART dataset, with a factually inconsistent output from the Pythia-1.4B model.
• Figure 2: All three stages of our statistical validation framework.
• Figure 3: Visualization of exponential and power law scaling of factual inconsistency (AlignScore) across datasets and LLM families, with margin of error (MOE) and $95\%$ confidence intervals on residuals.
• Figure 4: Visualization of exponential and power law scaling of factual inconsistency (QAFactEval) across datasets and LLM families, with margin of error (MOE) and $95\%$ confidence intervals on residuals.
• Figure 5: Visualization of exponential and power law scaling of factual inconsistency (SummaC-conv) across datasets and LLM families, with margin of error (MOE) and $95\%$ confidence intervals on residuals.