TaTa: A Multilingual Table-to-Text Dataset for African Languages

TL;DR

TaTA addresses the scarcity of multilingual data-to-text resources by introducing a large parallel table-to-text dataset focused on African languages, built from DHS reports with professional translation. The work demonstrates that current models struggle to produce understandable and faithful outputs for TaTA, and existing metrics poorly reflect human judgments, prompting the development of the StATA metric for better evaluation. Through monolingual, cross-lingual, and multilingual experiments using mT5 variants, TaTA reveals surprising cross-lingual transfer patterns (notably Swahili) and highlights the need for robust evaluation in low-resource languages, including zero-shot settings. The paper releases all data and annotations, provides a detailed analysis of model performance and evaluation limitations, and sets the stage for future improvements in cross-lingual data-to-text and multilingual metric design.

Abstract

Existing data-to-text generation datasets are mostly limited to English. To address this lack of data, we create Table-to-Text in African languages (TaTa), the first large multilingual table-to-text dataset with a focus on African languages. We created TaTa by transcribing figures and accompanying text in bilingual reports by the Demographic and Health Surveys Program, followed by professional translation to make the dataset fully parallel. TaTa includes 8,700 examples in nine languages including four African languages (Hausa, Igbo, Swahili, and Yorùbá) and a zero-shot test language (Russian). We additionally release screenshots of the original figures for future research on multilingual multi-modal approaches. Through an in-depth human evaluation, we show that TaTa is challenging for current models and that less than half the outputs from an mT5-XXL-based model are understandable and attributable to the source data. We further demonstrate that existing metrics perform poorly for TaTa and introduce learned metrics that achieve a high correlation with human judgments. We release all data and annotations at https://github.com/google-research/url-nlp.

Figures (6)

• Figure 1: An example from TaTA, which demonstrates many of the reasoning challenges it poses.
• Figure 2: An example of the process from infographic to linearized input. Each table value is encoded into a triple of (Column, Row, Value). The goal of the model is to generate text similar to the references below.
• Figure 3: Cross-lingual zero-shot transfer performance of different monolingual models across all language pairs. Each value represents the StATA QE metric for an XXL model trained on one language (rows) and evaluated on another one (columns). The final row/column represent an average.
• Figure 4: Two example info-graphics and their associated descriptions. Colored rectangles indicate where information from the text can be found in the figure. (A) The first sentence compares all numbers except the aggregate and infers that the numbers are increasing. The second sentence does not require any reasoning, but requires the inference that 6--59 months can be stated as "under the age of 5". (B) This sentence requires identifying the overall trend and calculating the peak population increase as the difference between birth and death rate ($31.9-6.5=25.5$). In addition to the values, sentences across both examples require accessing the title, unit of measure, or axis labels.
• Figure 5: Cross-lingual zero-shot transfer performance of different monolingual models across all language pairs using standard metrics. Each value represents an average over the traditional metrics for a model trained on one language (rows) and evaluated on another one (columns). The final row/column represent an average. As expected, the highest values are along the within-language diagonal, but we also observe some curious behavior for Hausa and Yorùbá and in general large disagreements with the numbers presented in Figure \ref{['fig:transfer-all']}.