BiVert: Bidirectional Vocabulary Evaluation using Relations for Machine Translation

TL;DR

A bidirectional semantic-based evaluation method designed to assess the sense distance of the translation from the source text, which introduces a quantifiable approach that empowers sentence comparison on the same linguistic level.

Abstract

Neural machine translation (NMT) has progressed rapidly in the past few years, promising improvements and quality translations for different languages. Evaluation of this task is crucial to determine the quality of the translation. Overall, insufficient emphasis is placed on the actual sense of the translation in traditional methods. We propose a bidirectional semantic-based evaluation method designed to assess the sense distance of the translation from the source text. This approach employs the comprehensive multilingual encyclopedic dictionary BabelNet. Through the calculation of the semantic distance between the source and its back translation of the output, our method introduces a quantifiable approach that empowers sentence comparison on the same linguistic level. Factual analysis shows a strong correlation between the average evaluation scores generated by our method and the human assessments across various machine translation systems for English-German language pair. Finally, our method proposes a new multilingual approach to rank MT systems without the need for parallel corpora.

Figures (5)

• Figure 1: Example of a direct translation from English to Russian using the system we wish to evaluate, and its back-translation using a state-of-the-art translation system suitable for BiVert.
• Figure 2: An example of final words alignment using the linear sum assignment problem algorithm.
• Figure 3: Example of words inconsequential and unimportant with illustrative embedding values, demonstrating different subword pooling strategies for word alignment. The word alignment algorithm calculates the cosine similarity between the embeddings representing the words chosen via option 1 or 2.
• Figure 4: Fragment of a semantic graph between the two words challenge and problem. The hatched grey edges connect roots to their senses, and the red edges represent hypernym relations between the nodes contents.
• Figure 5: A comparison of average human scores and average BiVert scores for each language pair on all translation systems.