Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Challenge of Achieving Attributability in Multilingual Table-to-Text Generation with Question-Answer Blueprints

Aden Haussmann

TL;DR

The paper investigates whether QA blueprints can improve the attributability of multilingual Table-to-Text verbalisation. By extending the TaTA dataset with QA blueprints and finetuning mT5 variants, it demonstrates clear gains in English but limited improvements in multilingual settings due to translation errors and weaker alignment between blueprints and generated text. It introduces and evaluates metrics (chrF, BLEU, FactKB, StATA) and finds StATA the most reliable automatic measure for TaTA attributability, while highlighting significant multilingual challenges. The findings suggest that larger multilingual data, constrained decoding, and more language-aware blueprint strategies are needed to realize attributability gains across languages, and it releases tools to facilitate further TaTA research.

Abstract

Multilingual Natural Language Generation (NLG) is challenging due to the lack of training data for low-resource languages. However, some low-resource languages have up to tens of millions of speakers globally, making it important to improve NLG tools for them. Table-to-Text NLG is an excellent measure of models' reasoning abilities but is very challenging in the multilingual setting. System outputs are often not attributable, or faithful, to the data in the source table. Intermediate planning techniques like Question-Answer (QA) blueprints have been shown to improve attributability on summarisation tasks. This work explores whether QA blueprints make multilingual Table-to-Text outputs more attributable to the input tables. This paper extends the challenging multilingual Table-to-Text dataset, TaTA, which includes African languages, with QA blueprints. Sequence-to-sequence language models are then finetuned on this dataset, with and without blueprints. Results show that QA blueprints improve performance for models finetuned and evaluated only on English examples, but do not demonstrate gains in the multilingual setting. This is due to inaccuracies in machine translating the blueprints from English into target languages when generating the training data, and models failing to rely closely on the blueprints they generate. An in-depth analysis is conducted on why this is challenging.

The Challenge of Achieving Attributability in Multilingual Table-to-Text Generation with Question-Answer Blueprints

TL;DR

The paper investigates whether QA blueprints can improve the attributability of multilingual Table-to-Text verbalisation. By extending the TaTA dataset with QA blueprints and finetuning mT5 variants, it demonstrates clear gains in English but limited improvements in multilingual settings due to translation errors and weaker alignment between blueprints and generated text. It introduces and evaluates metrics (chrF, BLEU, FactKB, StATA) and finds StATA the most reliable automatic measure for TaTA attributability, while highlighting significant multilingual challenges. The findings suggest that larger multilingual data, constrained decoding, and more language-aware blueprint strategies are needed to realize attributability gains across languages, and it releases tools to facilitate further TaTA research.

Abstract

Multilingual Natural Language Generation (NLG) is challenging due to the lack of training data for low-resource languages. However, some low-resource languages have up to tens of millions of speakers globally, making it important to improve NLG tools for them. Table-to-Text NLG is an excellent measure of models' reasoning abilities but is very challenging in the multilingual setting. System outputs are often not attributable, or faithful, to the data in the source table. Intermediate planning techniques like Question-Answer (QA) blueprints have been shown to improve attributability on summarisation tasks. This work explores whether QA blueprints make multilingual Table-to-Text outputs more attributable to the input tables. This paper extends the challenging multilingual Table-to-Text dataset, TaTA, which includes African languages, with QA blueprints. Sequence-to-sequence language models are then finetuned on this dataset, with and without blueprints. Results show that QA blueprints improve performance for models finetuned and evaluated only on English examples, but do not demonstrate gains in the multilingual setting. This is due to inaccuracies in machine translating the blueprints from English into target languages when generating the training data, and models failing to rely closely on the blueprints they generate. An in-depth analysis is conducted on why this is challenging.

Paper Structure

This paper contains 28 sections, 2 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. QA Blueprint Creation
  4. Extracting propositions
  5. Generating & Heuristically Selecting QA Pairs
  6. Building the Blueprint
  7. Making Blueprints Multilingual
  8. Repetition Penalty
  9. Automatic Evaluation
  10. chrF & BLEU
  11. FactKB
  12. StATA
  13. Results
  14. English Subset Results
  15. Examples & common patterns
  16. ...and 13 more sections

Figures (4)

  • Figure 1: An example of a chart image, its linearised form (created by human annotators), and an output verbalisation (produced by a seq2seq neural network).
  • Figure 2: The process of generating QA blueprints.
  • Figure 3: Pearson correlations of mT5-Small and mT5-Large StATA models with U+A human evaluations.
  • Figure 4: Multilingual model performance on the test set (including all languages) with varying repetition penalties.