Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Qomhra: A Bilingual Irish-English Large Language Model

Joseph McInerney

TL;DR

Qomhrá addresses the data scarcity of Irish language resources by developing a bilingual Irish-English LLM through bilingual continued pre-training, instruction tuning, and human preference alignment. The approach mixes new Irish data with English data on a Qwen3-8B-Base foundation, uses synthetic data generated by closed-weight LLMs with Gemini-2.5-Pro identified as the strongest Irish generator, and creates a 30K Irish-English parallel instruction tuning dataset plus a 1K human preference set. Instruction tuning with the best synthetic data followed by a native-aligned human feedback phase yields near perfect alignment in preference data and substantial benchmark gains, including up to 29% improvement in Irish and 44% in English on IQA. The work demonstrates that synthetic data and careful cross-language pre-training can enable practical Irish chatbot capabilities, while acknowledging limitations such as data scarcity, reliance on a single native annotator, and potential biases from translated instruction data.

Abstract

This paper introduces Qomhrá, a bilingual Irish-English large language model (LLM), developed under low-resource constraints presenting a complete pipeline spanning bilingual continued pre-training, instruction tuning, and alignment from human preferences. Newly accessible Irish corpora and English text are mixed and curated to improve Irish performance while preserving English ability. 6 closed-weight LLMs are judged for their Irish text generation by a native speaker, a learner and other LLMs. Google's Gemini-2.5-Pro is ranked the highest and is subsequently used to synthesise instruction tuning and human preference datasets. Two datasets are contributed leveraging Gemini-2.5-Pro: a 30K Irish-English parallel instruction tuning dataset and a 1K human preference dataset, generating accepted and rejected responses that show near perfect alignment with a native Irish speaker. Qomhrá is comprehensively evaluated across benchmarks testing translation, gender understanding, topic identification and world knowledge with gains of up to 29% in Irish and 44% in English. Qomhrá also undergoes instruction tuning and demonstrates clear progress in instruction following, crucial for chatbot functionality.

Qomhra: A Bilingual Irish-English Large Language Model

TL;DR

Qomhrá addresses the data scarcity of Irish language resources by developing a bilingual Irish-English LLM through bilingual continued pre-training, instruction tuning, and human preference alignment. The approach mixes new Irish data with English data on a Qwen3-8B-Base foundation, uses synthetic data generated by closed-weight LLMs with Gemini-2.5-Pro identified as the strongest Irish generator, and creates a 30K Irish-English parallel instruction tuning dataset plus a 1K human preference set. Instruction tuning with the best synthetic data followed by a native-aligned human feedback phase yields near perfect alignment in preference data and substantial benchmark gains, including up to 29% improvement in Irish and 44% in English on IQA. The work demonstrates that synthetic data and careful cross-language pre-training can enable practical Irish chatbot capabilities, while acknowledging limitations such as data scarcity, reliance on a single native annotator, and potential biases from translated instruction data.

Abstract

This paper introduces Qomhrá, a bilingual Irish-English large language model (LLM), developed under low-resource constraints presenting a complete pipeline spanning bilingual continued pre-training, instruction tuning, and alignment from human preferences. Newly accessible Irish corpora and English text are mixed and curated to improve Irish performance while preserving English ability. 6 closed-weight LLMs are judged for their Irish text generation by a native speaker, a learner and other LLMs. Google's Gemini-2.5-Pro is ranked the highest and is subsequently used to synthesise instruction tuning and human preference datasets. Two datasets are contributed leveraging Gemini-2.5-Pro: a 30K Irish-English parallel instruction tuning dataset and a 1K human preference dataset, generating accepted and rejected responses that show near perfect alignment with a native Irish speaker. Qomhrá is comprehensively evaluated across benchmarks testing translation, gender understanding, topic identification and world knowledge with gains of up to 29% in Irish and 44% in English. Qomhrá also undergoes instruction tuning and demonstrates clear progress in instruction following, crucial for chatbot functionality.
Paper Structure (30 sections, 1 equation, 2 figures, 5 tables)

This paper contains 30 sections, 1 equation, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. CPT
  5. Irish Text Collection
  6. Pre-processing
  7. Deduplication
  8. Segmentation
  9. Vocabulary Expansion
  10. Evaluation
  11. Training Configuration and Hyper-parameters
  12. Instruction Tuning
  13. Training Configuration and Hyper-parameters
  14. Human Feedback
  15. Experimentation and Results
  16. ...and 15 more sections

Figures (2)

  • Figure 1: High-Level Pipeline Overview
  • Figure 2: Inter-Annotator Agreement