Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Y-NQ: English-Yorùbá Evaluation dataset for Open-Book Reading Comprehension and Text Generation

Marta R. Costa-jussà, Joy Chen, Ifeoluwanimi Adebara, Joe Chuang, Christophe Ropers, Eduardo Sánchez

TL;DR

Y-NQ introduces a bilingual, open-book reading comprehension and text-generation dataset derived from Natural Questions to benchmark large language models on English versus Yorùbá. The authors collect and annotate cross-language pairs, report a persistent English advantage on automatic metrics, and show that Yorùbá documents—though shorter—pose greater long-context challenges for current models. Their analysis includes baseline evaluations with GPT-4o, o1-mini, and LLaMA-3.1-8b, and a detailed examination of length effects and data quality, including cross-language annotation discrepancies. The dataset, released on HuggingFace, provides a resource for assessing cross-language RC capabilities and highlights limitations in extending English-language reading comprehension to Yorùbá, with implications for multilingual NLP research and model development.

Abstract

The purpose of this work is to share an English-Yorùbá evaluation dataset for open-book reading comprehension and text generation to assess the performance of models both in a high- and a low- resource language. The dataset contains 358 questions and answers on 338 English documents and 208 Yorùbá documents. The average document length is ~ 10k words for English and 430 words for Yorùbá. Experiments show a consistent disparity in performance between the two languages, with Yorùbá falling behind English for automatic metrics even if documents are much shorter for this language. For a small set of documents with comparable length, performance of Yorùbá drops by x2.5 times. When analyzing performance by length, we observe that Yorùbá decreases performance dramatically for documents that reach 1500 words while English performance is barely affected at that length. Our dataset opens the door to showcasing if English LLM reading comprehension capabilities extend to Yorùbá, which for the evaluated LLMs is not the case.

Y-NQ: English-Yorùbá Evaluation dataset for Open-Book Reading Comprehension and Text Generation

TL;DR

Y-NQ introduces a bilingual, open-book reading comprehension and text-generation dataset derived from Natural Questions to benchmark large language models on English versus Yorùbá. The authors collect and annotate cross-language pairs, report a persistent English advantage on automatic metrics, and show that Yorùbá documents—though shorter—pose greater long-context challenges for current models. Their analysis includes baseline evaluations with GPT-4o, o1-mini, and LLaMA-3.1-8b, and a detailed examination of length effects and data quality, including cross-language annotation discrepancies. The dataset, released on HuggingFace, provides a resource for assessing cross-language RC capabilities and highlights limitations in extending English-language reading comprehension to Yorùbá, with implications for multilingual NLP research and model development.

Abstract

The purpose of this work is to share an English-Yorùbá evaluation dataset for open-book reading comprehension and text generation to assess the performance of models both in a high- and a low- resource language. The dataset contains 358 questions and answers on 338 English documents and 208 Yorùbá documents. The average document length is ~ 10k words for English and 430 words for Yorùbá. Experiments show a consistent disparity in performance between the two languages, with Yorùbá falling behind English for automatic metrics even if documents are much shorter for this language. For a small set of documents with comparable length, performance of Yorùbá drops by x2.5 times. When analyzing performance by length, we observe that Yorùbá decreases performance dramatically for documents that reach 1500 words while English performance is barely affected at that length. Our dataset opens the door to showcasing if English LLM reading comprehension capabilities extend to Yorùbá, which for the evaluated LLMs is not the case.

Paper Structure

This paper contains 17 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Dataset description
  3. Requirements and Background
  4. Dataset.
  5. Low-resource language.
  6. Dataset creation
  7. NQ pre-selection.
  8. Pre-annotation effort.
  9. Annotation guidelines and requirements.
  10. Annotator findings.
  11. Statistics.
  12. Experiments
  13. Baselines
  14. Evaluation.
  15. Automatic metrics.
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Impact of Document Length Buckets on Performance Scores for English (top) and Yorùbá (bottom) for GPT-4 outputs