Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo

Barkavi Sundararajan, Somayajulu Sripada, Ehud Reiter

TL;DR

This work investigates why neural table-to-text systems hallucinate by tracing errors to input data problems in the ToTTo politics domain. It demonstrates that systematic input corrections—addressing non-atomic cells, missing values, complex headers, and domain-specific semantics—substantially reduce factual errors, with reductions up to 76% for Llama 2-13B and 62% for T5-base on corrected data. The study employs meticulous manual error annotation with solid inter-annotator agreement to categorize faults and validate the causal link between input quality and output fidelity. While improvements are clear, some omissions persist and vary by model, suggesting the need for broader datasets and further refinement of input-correction strategies. The findings highlight a practical path to improving factual accuracy in tabular-to-text generation by prioritizing input integrity, potentially benefiting real-world applications that rely on faithful data-to-language translation.

Abstract

Neural Table-to-Text models tend to hallucinate, producing texts that contain factual errors. We investigate whether such errors in the output can be traced back to problems with the input. We manually annotated 1,837 texts generated by multiple models in the politics domain of the ToTTo dataset. We identify the input problems that are responsible for many output errors and show that fixing these inputs reduces factual errors by between 52% and 76% (depending on the model). In addition, we observe that models struggle in processing tabular inputs that are structured in a non-standard way, particularly when the input lacks distinct row and column values or when the column headers are not correctly mapped to corresponding values.

Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo

TL;DR

This work investigates why neural table-to-text systems hallucinate by tracing errors to input data problems in the ToTTo politics domain. It demonstrates that systematic input corrections—addressing non-atomic cells, missing values, complex headers, and domain-specific semantics—substantially reduce factual errors, with reductions up to 76% for Llama 2-13B and 62% for T5-base on corrected data. The study employs meticulous manual error annotation with solid inter-annotator agreement to categorize faults and validate the causal link between input quality and output fidelity. While improvements are clear, some omissions persist and vary by model, suggesting the need for broader datasets and further refinement of input-correction strategies. The findings highlight a practical path to improving factual accuracy in tabular-to-text generation by prioritizing input integrity, potentially benefiting real-world applications that rely on faithful data-to-language translation.

Abstract

Neural Table-to-Text models tend to hallucinate, producing texts that contain factual errors. We investigate whether such errors in the output can be traced back to problems with the input. We manually annotated 1,837 texts generated by multiple models in the politics domain of the ToTTo dataset. We identify the input problems that are responsible for many output errors and show that fixing these inputs reduces factual errors by between 52% and 76% (depending on the model). In addition, we observe that models struggle in processing tabular inputs that are structured in a non-standard way, particularly when the input lacks distinct row and column values or when the column headers are not correctly mapped to corresponding values.
Paper Structure (43 sections, 2 figures, 14 tables)

This paper contains 43 sections, 2 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Table to Text dataset, ToTTo
  3. Related Work
  4. Pilot Study
  5. Methodology
  6. Insights from our Pilot Study
  7. Input Problems
  8. Generic Input Problems
  9. Politics Domain-Specific Input Problems
  10. Main Study
  11. Llama 2 Models
  12. Manual Fixes for Input Problems
  13. Non-Atomic Cell Values
  14. Complex Table Type
  15. Insufficient Input
  16. ...and 28 more sections

Figures (2)

  • Figure 1: Model Specifications
  • Figure 2: Comparison of different input problem types before (original data) and after data correction across four models. It shows the count of unique samples for three categories: (i) errors, (ii) omissions, and (iii) no errors.