SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window

TL;DR

The paper addresses the insufficiency of sentence-level evaluation for MT by proposing SLIDE, a sliding-document-window method that augments a fixed-context quality estimator (e.g., COMET-QE with InfoXLM) with document-level context. By using a strided window of width $w$ and stride $s$, SLIDE feeds multi-sentence chunks to an unmodified QE model and averages chunk scores to obtain document- and system-level assessments. The study shows that source context can substitute for references in disambiguation, yielding higher pairwise system accuracy and sometimes matching reference-based metrics, notably on the WMT22 MQM/DA+SQM tasks. This approach enables higher-quality, reference-free document-level evaluation with minimal changes to existing QE tools, contingent on the availability of document boundary annotations.

Abstract

Reference-based metrics that operate at the sentence-level typically outperform quality estimation metrics, which have access only to the source and system output. This is unsurprising, since references resolve ambiguities that may be present in the source. In this paper, we investigate whether additional source context can effectively substitute for a reference. We present a metric named SLIDE (SLIding Document Evaluator), which operates on blocks of sentences. SLIDE leverages a moving window that slides over each document in the test set, feeding each chunk of sentences into an unmodified, off-the-shelf quality estimation model. We find that SLIDE obtains significantly higher pairwise system accuracy than its sentence-level baseline, in some cases even eliminating the gap with reference-base metrics. This suggests that source context may provide the same information as a human reference in disambiguating source ambiguities. This finding is especially pertinent for reference-free document-level evaluation, wherein SLIDE could provide higher-quality pairwise system assessments while only requiring document boundary annotations.

Figures (4)

• Figure 1: SLIDE extraction for $(w=4,s=2)$. The solid green boxes denote extracted chunks, which are then joined with a space and sent to COMET as a single unit. The dashed red boxes denote partial documents: a document that is too short (top), and a document remainder (bottom).
• Figure 2: Plot of window vs. stride accuracies for QE20, QE22, and ®COMET22 models on the WMT22 MQM task (en-de, en-ru, and zh-en). Neither COMET20-QE nor the reference-based ®COMET22 sees much improvement from adding context, but the QE22 model does.
• Figure 3: Percentage of original-document sentences that are dropped when partial documents are ignored.
• Figure 4: Heatmaps of MQM accuracy difference relative to Figure \ref{['figure:heatmap']}(b) for SLIDE when all partials are incorporated. In (a), they are treated as equal chunks when producing the document-level score; in (b), the score of each chunk is upweighted based on the number of sentences it contains. Both variants improve over context-less QE22, but are generally worse than SLIDE$_f$