Investigating Length Issues in Document-level Machine Translation

TL;DR

The paper investigates length-related degradation in document-level machine translation by designing a controlled methodology that varies input length and sentence position, evaluated on two architectures with doc-to-doc translation. It introduces ds-BLEU and a paired testing framework to quantify length and position effects, and proposes unifPE to balance training exposure to positional encodings. Key findings show that translation quality generally declines as document length grows, with earlier sentences translating more reliably than later ones; distributing or reshaping positional encoding exposure yields model-dependent improvements, notably for encoder-decoder NLLB-based systems, while RoPE-based TowerBase remains less responsive. The work reinforces that sentence-level MT remains a strong baseline and highlights avenues for improving document-level MT, such as attention constraints to mimic sentence alignment or better memorization strategies, with broader implications for evaluation and training practices in long-context NLP.

Abstract

Transformer architectures are increasingly effective at processing and generating very long chunks of texts, opening new perspectives for document-level machine translation (MT). In this work, we challenge the ability of MT systems to handle texts comprising up to several thousands of tokens. We design and implement a new approach designed to precisely measure the effect of length increments on MT outputs. Our experiments with two representative architectures unambiguously show that (a)~translation performance decreases with the length of the input text; (b)~the position of sentences within the document matters, and translation quality is higher for sentences occurring earlier in a document. We further show that manipulating the distribution of document lengths and of positional embeddings only marginally mitigates such problems. Our results suggest that even though document-level MT is computationally feasible, it does not yet match the performance of sentence-based MT.

Figures (3)

• Figure 1: Top: probability of observing training position $i$ ($P(i)$) for a sentence of length $l=200$, with standard training ($k_i=0$, left) and with our uniform sampling scheme (right) for $M = 512$. Middle: original, unifPE, and SHAPE$P(i)$ for training set TED-G and $M = 2048$. Bottom: original, unifPE and SHAPE$P(i)$ for TED-U and $M = 2048$.
• Figure 2: BLEU, ds-BLEU and d-BLEU scores for IWSLT tst2015, translating and evaluating pseudo-documents of increasing lengths $[256, 512, 768, 1204]$, using FT-NLLB-U (top) and Unif-Tower-U (bottom). Note that d-BLEU is computed for pseudo-documents while ds-BLEU is computed for concatenated full talks.
• Figure 3: Source document length distribution in the training set of TED-G (top) and TED-U (bottom).