Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Syntax-Aware Complex-Valued Neural Machine Translation

Yang Liu, Yuexian Hou

TL;DR

This work proposes a method to incorporate syntax information into a complex-valued Encoder-Decoder architecture that jointly learns word-level and syntax-level attention scores from the source side to the target side using an attention mechanism.

Abstract

Syntax has been proven to be remarkably effective in neural machine translation (NMT). Previous models obtained syntax information from syntactic parsing tools and integrated it into NMT models to improve translation performance. In this work, we propose a method to incorporate syntax information into a complex-valued Encoder-Decoder architecture. The proposed model jointly learns word-level and syntax-level attention scores from the source side to the target side using an attention mechanism. Importantly, it is not dependent on specific network architectures and can be directly integrated into any existing sequence-to-sequence (Seq2Seq) framework. The experimental results demonstrate that the proposed method can bring significant improvements in BLEU scores on two datasets. In particular, the proposed method achieves a greater improvement in BLEU scores in translation tasks involving language pairs with significant syntactic differences.

Syntax-Aware Complex-Valued Neural Machine Translation

TL;DR

This work proposes a method to incorporate syntax information into a complex-valued Encoder-Decoder architecture that jointly learns word-level and syntax-level attention scores from the source side to the target side using an attention mechanism.

Abstract

Syntax has been proven to be remarkably effective in neural machine translation (NMT). Previous models obtained syntax information from syntactic parsing tools and integrated it into NMT models to improve translation performance. In this work, we propose a method to incorporate syntax information into a complex-valued Encoder-Decoder architecture. The proposed model jointly learns word-level and syntax-level attention scores from the source side to the target side using an attention mechanism. Importantly, it is not dependent on specific network architectures and can be directly integrated into any existing sequence-to-sequence (Seq2Seq) framework. The experimental results demonstrate that the proposed method can bring significant improvements in BLEU scores on two datasets. In particular, the proposed method achieves a greater improvement in BLEU scores in translation tasks involving language pairs with significant syntactic differences.
Paper Structure (16 sections, 19 equations, 5 figures, 2 tables)

This paper contains 16 sections, 19 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Neural Machine Translation
  4. Complex-valued Neural Network
  5. Proposed Approach
  6. Syntax Embedding
  7. Complex-valued Neural Machine Translation
  8. Syntax-based Loss Function
  9. Experiments
  10. Settings
  11. Baseline Systems
  12. Evaluating SynCoNMT
  13. Effect of Translating Long Sentences
  14. Complex-valued Attention
  15. Related Work
  16. ...and 1 more sections

Figures (5)

  • Figure 1: An example that illustrates how different syntactic dependency structures can lead to a change in the meaning of a sentence.
  • Figure 2: An example that illustrates how the same sentence meaning can have different syntactic dependency structures in different languages.
  • Figure 3: Syntax-Aware Complex-Valued Neural Machine Translation. The words and their syntactic information are represented as complex-valued vectors with real and imaginary components respectively, and the entire network is computed using complex-valued operations.
  • Figure 4: The performance of SynCoNMT on different sentence lengths.
  • Figure 5: The x-axis and y-axis of each figure respectively represent the words or dependencies in the target language (English) and the source language (Chinese). (a) shows the results of traditional attention. (b) shows the real part of attention scores in SynCoNMT. (c) shows the imaginary part of attention scores in SynCoNMT.