Word Closure-Based Metamorphic Testing for Machine Translation

TL;DR

This work tackles the oracle problem in machine translation testing by introducing a word-closure–based output comparison framework that enables fine-grained, semantics-aware violations detection. By linking corresponding fragments across source and follow-up translations and evaluating semantic similarity at the word-closure level, the method overcomes coarse-grained and loosely linked comparisons that hinder prior metamorphic testing approaches. The framework builds word closures via refined word alignment, constructs closures to cover translation tokens, and classifies comparisons into CWCs, MWCs, and UWCs for robust MR-based testing across five input transformations. Empirical results across Google, Bing, and Youdao in English-Chinese and Chinese-English tasks show substantial gains in violation identification (average F1 up to 29.9%) and fine-grained violation localization (up to 35.9%), along with best performance achieved using a hybrid semantic similarity configuration. The work also discusses practical implications, efficiency, and directions for future improvements in MT testing and repair workflows.

Abstract

With the wide application of machine translation, the testing of Machine Translation Systems (MTSs) has attracted much attention. Recent works apply Metamorphic Testing (MT) to address the oracle problem in MTS testing. Existing MT methods for MTS generally follow the workflow of input transformation and output relation comparison, which generates a follow-up input sentence by mutating the source input and compares the source and follow-up output translations to detect translation errors, respectively. These methods use various input transformations to generate test case pairs and have successfully triggered numerous translation errors. However, they have limitations in performing fine-grained and rigorous output relation comparison and thus may report many false alarms and miss many true errors. In this paper, we propose a word closure-based output comparison method to address the limitations of the existing MTS MT methods. We first propose word closure as a new comparison unit, where each closure includes a group of correlated input and output words in the test case pair. Word closures suggest the linkages between the appropriate fragment in the source output translation and its counterpart in the follow-up output for comparison. Next, we compare the semantics on the level of word closure to identify the translation errors. In this way, we perform a fine-grained and rigorous semantic comparison for the outputs and thus realize more effective violation identification. We evaluate our method with the test cases generated by five existing input transformations and the translation outputs from three popular MTSs. Results show that our method significantly outperforms the existing works in violation identification by improving the precision and recall and achieving an average increase of 29.9% in F1 score. It also helps to increase the F1 score of translation error localization by 35.9%.

Figures (17)

• Figure 1: Examples of the limitations of the existing output comparison methods.
• Figure 2: Overview of our approach.
• Figure 3: Matching between $T_s$ and $T_f$ in PatInv-FN at different granularities. The box represents the granularity of clause, phrase, word, or word closure, in which the blue ones can be matched between $T_s$ and $T_f$. We subdivide the translation into clauses and phrases based on its constituency parse tree generated by Stanford Parser stanford-parser, where "IP" denotes a clause and "NP", "DNP", "PP", "VP" refer to various types of phrases.
• Figure 4: Word alignment refinement Algorithm \ref{['alg:one']} on PatInv-FN.
• Figure 5: Word alignment refinement Algorithm \ref{['alg:two']} on CIT-FN.

Theorems & Definitions (1)

• Definition 1: Word Closure