Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M3T: A New Benchmark Dataset for Multi-Modal Document-Level Machine Translation

Benjamin Hsu, Xiaoyu Liu, Huayang Li, Yoshinari Fujinuma, Maria Nadejde, Xing Niu, Yair Kittenplon, Ron Litman, Raghavendra Pappagari

TL;DR

The paper tackles translating visually rich, semi-structured PDF documents by arguing that traditional sentence-level MT misses essential document context embodied in layout and reading order. It introduces M3T, a benchmark for end-to-end Multi-Modal MT of semi-structured PDFs, with layout and reading-order annotations and supplementary synthetic data drawn from EUR-Lex, DocLayNet, and RVL-CDIP. Using a multi-modal foundation model (LLaVa-v1.5), it shows that incorporating visual context can mitigate OCR errors and provide document-level cues, as reflected in doc-COMET scores, though gains are varied and current models still struggle with consistency and hallucinations. The dataset and baseline findings aim to spur progress in robust multi-modal document translation and provide resources for future research on alignment, prompts, and model tuning.

Abstract

Document translation poses a challenge for Neural Machine Translation (NMT) systems. Most document-level NMT systems rely on meticulously curated sentence-level parallel data, assuming flawless extraction of text from documents along with their precise reading order. These systems also tend to disregard additional visual cues such as the document layout, deeming it irrelevant. However, real-world documents often possess intricate text layouts that defy these assumptions. Extracting information from Optical Character Recognition (OCR) or heuristic rules can result in errors, and the layout (e.g., paragraphs, headers) may convey relationships between distant sections of text. This complexity is particularly evident in widely used PDF documents, which represent information visually. This paper addresses this gap by introducing M3T, a novel benchmark dataset tailored to evaluate NMT systems on the comprehensive task of translating semi-structured documents. This dataset aims to bridge the evaluation gap in document-level NMT systems, acknowledging the challenges posed by rich text layouts in real-world applications.

M3T: A New Benchmark Dataset for Multi-Modal Document-Level Machine Translation

TL;DR

The paper tackles translating visually rich, semi-structured PDF documents by arguing that traditional sentence-level MT misses essential document context embodied in layout and reading order. It introduces M3T, a benchmark for end-to-end Multi-Modal MT of semi-structured PDFs, with layout and reading-order annotations and supplementary synthetic data drawn from EUR-Lex, DocLayNet, and RVL-CDIP. Using a multi-modal foundation model (LLaVa-v1.5), it shows that incorporating visual context can mitigate OCR errors and provide document-level cues, as reflected in doc-COMET scores, though gains are varied and current models still struggle with consistency and hallucinations. The dataset and baseline findings aim to spur progress in robust multi-modal document translation and provide resources for future research on alignment, prompts, and model tuning.

Abstract

Document translation poses a challenge for Neural Machine Translation (NMT) systems. Most document-level NMT systems rely on meticulously curated sentence-level parallel data, assuming flawless extraction of text from documents along with their precise reading order. These systems also tend to disregard additional visual cues such as the document layout, deeming it irrelevant. However, real-world documents often possess intricate text layouts that defy these assumptions. Extracting information from Optical Character Recognition (OCR) or heuristic rules can result in errors, and the layout (e.g., paragraphs, headers) may convey relationships between distant sections of text. This complexity is particularly evident in widely used PDF documents, which represent information visually. This paper addresses this gap by introducing M3T, a novel benchmark dataset tailored to evaluate NMT systems on the comprehensive task of translating semi-structured documents. This dataset aims to bridge the evaluation gap in document-level NMT systems, acknowledging the challenges posed by rich text layouts in real-world applications.
Paper Structure (11 sections, 3 figures, 5 tables)

This paper contains 11 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Dataset Description
  3. Data Sourcing
  4. Document Annotation
  5. Supplementary Data
  6. Multi-Modal Document Translation
  7. Results
  8. Conclusions
  9. Limitations
  10. Annotation Details
  11. Detailed Results

Figures (3)

  • Figure 1: Example OCR errors that occur in layout-agnostic systems. Reading order is lost when extracting text and column breaks are not correctly recognized.
  • Figure 2: We introduce a benchmark dataset for PDF document translation. Unlike earlier works, our benchmark that focus on sentence-level text or cleanly segmented text, ours tests models on their ability to utilize visual queues for translating text. Above from left to right, are example documents sourced from the RVL-CDIP and DocLaynet datasets demonstrating the complex layout and domains included in the benchmark. Additional examples can be found in the Appendix, Figure \ref{['fig:example_annotations_app']}.
  • Figure 3: In this work, we introduce a benchmark dataset for PDF document translation. Example documents sourced from (a) EUR-Lex (b) DocLayNet and (c) RVL-CDIP datasets demonstrating the complex layout and domains included in the benchmark. Unlike earlier works, our benchmark focuses on machine translation which depends more heavily on contextual information than earlier works centered around entity extraction.