Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Open Machine Translation for Esperanto

Ona de Gibert, Lluís de Gibert

Abstract

Esperanto is a widespread constructed language, known for its regular grammar and productive word formation. Besides having substantial resources available thanks to its online community, it remains relatively underexplored in the context of modern machine translation (MT) approaches. In this work, we present the first comprehensive evaluation of open-source MT systems for Esperanto, comparing rule-based systems, encoder-decoder models, and LLMs across model sizes. We evaluate translation quality across six language directions involving English, Spanish, Catalan, and Esperanto using multiple automatic metrics as well as human evaluation. Our results show that the NLLB family achieves the best performance in all language pairs, followed closely by our trained compact models and a fine-tuned general-purpose LLM. Human evaluation confirms this trend, with NLLB translations preferred in approximately half of the comparisons, although noticeable errors remain. In line with Esperanto's tradition of openness and international collaboration, we release our code and best-performing models publicly.

Open Machine Translation for Esperanto

Abstract

Esperanto is a widespread constructed language, known for its regular grammar and productive word formation. Besides having substantial resources available thanks to its online community, it remains relatively underexplored in the context of modern machine translation (MT) approaches. In this work, we present the first comprehensive evaluation of open-source MT systems for Esperanto, comparing rule-based systems, encoder-decoder models, and LLMs across model sizes. We evaluate translation quality across six language directions involving English, Spanish, Catalan, and Esperanto using multiple automatic metrics as well as human evaluation. Our results show that the NLLB family achieves the best performance in all language pairs, followed closely by our trained compact models and a fine-tuned general-purpose LLM. Human evaluation confirms this trend, with NLLB translations preferred in approximately half of the comparisons, although noticeable errors remain. In line with Esperanto's tradition of openness and international collaboration, we release our code and best-performing models publicly.

Paper Structure

This paper contains 33 sections, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. An Introduction to Esperanto
  3. Related Work
  4. Esperanto in NLP
  5. Esperanto in MT
  6. Experimental Setup
  7. Benchmarking
  8. Models
  9. Evaluation Data
  10. MT Development
  11. Training
  12. Data
  13. Vocabulary
  14. Evaluation Metrics
  15. Results
  16. ...and 18 more sections

Figures (4)

  • Figure 1: Cumulative number of ACL Anthology papers mentioning 'Esperanto' and 'Esperanto and Machine Translation'. Early work involving Esperanto focused primarily on MT, while more recent work covers a broader range of topics.
  • Figure 2: Prompt example for fine-tuning Llama-3.1-8B-Instruct
  • Figure 3: Human evaluation win rates for both translation directions.
  • Figure 4: Annotation guidelines shown to the human annotator.