Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FoQA: A Faroese Question-Answering Dataset

Annika Simonsen, Dan Saattrup Nielsen, Hafsteinn Einarsson

TL;DR

FoQA delivers the first Faroese extractive QA benchmark by introducing a semi-automated generation pipeline that combines GPT-4-turbo QA generation with native-speaker validation. It provides a SQuAD-like dataset released in three versions, and a comprehensive annotation and quality-control framework to ensure linguistic and contextual accuracy. Evaluations show decoder-based LLMs outperform encoder-based models on Faroese QA, underscoring the potential of large multilingual models for low-resource languages. The work also analyzes common error patterns and biases, and offers an open-source resource to propel future Faroese NLP research.

Abstract

We present FoQA, a Faroese extractive question-answering (QA) dataset with 2,000 samples, created using a semi-automated approach combining Large Language Models (LLMs) and human validation. The dataset was generated from Faroese Wikipedia articles using GPT-4-turbo for initial QA generation, followed by question rephrasing to increase complexity and native speaker validation to ensure quality. We provide baseline performance metrics for FoQA across multiple models, including LLMs and BERT, demonstrating its effectiveness in evaluating Faroese QA performance. The dataset is released in three versions: a validated set of 2,000 samples, a complete set of all 10,001 generated samples, and a set of 2,395 rejected samples for error analysis.

FoQA: A Faroese Question-Answering Dataset

TL;DR

FoQA delivers the first Faroese extractive QA benchmark by introducing a semi-automated generation pipeline that combines GPT-4-turbo QA generation with native-speaker validation. It provides a SQuAD-like dataset released in three versions, and a comprehensive annotation and quality-control framework to ensure linguistic and contextual accuracy. Evaluations show decoder-based LLMs outperform encoder-based models on Faroese QA, underscoring the potential of large multilingual models for low-resource languages. The work also analyzes common error patterns and biases, and offers an open-source resource to propel future Faroese NLP research.

Abstract

We present FoQA, a Faroese extractive question-answering (QA) dataset with 2,000 samples, created using a semi-automated approach combining Large Language Models (LLMs) and human validation. The dataset was generated from Faroese Wikipedia articles using GPT-4-turbo for initial QA generation, followed by question rephrasing to increase complexity and native speaker validation to ensure quality. We provide baseline performance metrics for FoQA across multiple models, including LLMs and BERT, demonstrating its effectiveness in evaluating Faroese QA performance. The dataset is released in three versions: a validated set of 2,000 samples, a complete set of all 10,001 generated samples, and a set of 2,395 rejected samples for error analysis.

Paper Structure

This paper contains 19 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Generation of Tentative Dataset
  5. Manual Filtering of Tentative Dataset
  6. Annotation Guidelines
  7. Tier 1: Grammatical Assessment
  8. Tier 2: Semantic and Contextual Assessment
  9. Tier 3: Final Classification
  10. Quality Control Process
  11. Faroese Setup
  12. The Dataset
  13. Format
  14. Statistics
  15. Dataset Versions
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Overview of the QA dataset generation pipeline. The system processes input documents to generate initial QA pairs, followed by a question rewriting phase that improves clarity while maintaining the original answers. All outputs follow a structured JSON format to ensure consistency. Note that while the outputs are in Faroese, the example shown in this figure uses an English example for illustrative purposes.