DOLFIN -- Document-Level Financial test set for Machine Translation

TL;DR

DOLFIN addresses the paucity of document-level MT benchmarks in the financial domain by introducing a section-based test set derived from Fundinfo documents, preserving structural elements via Markdown to support long-range context and information reorganization. It details a comprehensive processing pipeline that converts PDFs to section-aligned Markdown segments, applies extensive filtering and quality control, and labels context-sensitive phenomena to enable targeted evaluation. Experiments with large language models reveal that longer-context translations can improve quality for capable models (notably Llama-3-70b) but may not always help, highlighting the necessity of robust context handling in finance-specific translation. The resource is publicly released and provides a foundation for developing and meta-evaluating document-level metrics and context-aware translation approaches in specialized domains.

Abstract

Despite the strong research interest in document-level Machine Translation (MT), the test sets dedicated to this task are still scarce. The existing test sets mainly cover topics from the general domain and fall short on specialised domains, such as legal and financial. Also, in spite of their document-level aspect, they still follow a sentence-level logic that does not allow for including certain linguistic phenomena such as information reorganisation. In this work, we aim to fill this gap by proposing a novel test set: DOLFIN. The dataset is built from specialised financial documents, and it makes a step towards true document-level MT by abandoning the paradigm of perfectly aligned sentences, presenting data in units of sections rather than sentences. The test set consists of an average of 1950 aligned sections for five language pairs. We present a detailed data collection pipeline that can serve as inspiration for aligning new document-level datasets. We demonstrate the usefulness and quality of this test set by evaluating a number of models. Our results show that the test set is able to discriminate between context-sensitive and context-agnostic models and shows the weaknesses when models fail to accurately translate financial texts. The test set is made public for the community.